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NASA/Goddard Space Flight Center 
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November 24, 1980 


IN FRODUCTORY REMARKS F. McGarry/Goddard Space Flight Center 

PANEL NO. 1 ‘THE SOFTWARE ENGINEERING LABORATORY’’ 

DISCUSSANT: H. Hecht/SoHaR, Incorporated 


• F. McGarry (NASA/GSFC) 


• J. Page (CSC) 

• V. Basili (University of 
Maryland) 


“An Approach to Measuring Software 
Technology’’ 

“Impacts of Experiments and Software Tech- 
nology Changes in a Production Environment’’ 

“Measuring the Effects of Specific Software 
Methodologies Within the SEL” 


BREAK 

PANEL NO. 2 “SOFT^VARE COST/RESOURCE MODELING” 

DISCUSSANT: L. Putnam/Quantitative Software Management 

• J. Golden/J. Mueller/B. Anselm “Software Cost Estimating: Craft or 

(Xerox) Witchcraft” - 

• R. Tausworthe (JPL) “Deep Space Network Software Cost Estima- 

tion Model” 

• R. Lawler (Boeing) “Software Quality Tradeoff Measurement” 


LUNCH 


PANEL NO. 3 “SOFTWARE RELIABILITY” 

DISCUSSANT: J. Musa/Bell Laboratories 

• J, Logan (TRW) “Application of a Reliability Model to Require- 

ments Error Analysis” 

• A. Goel;J. Soenjoto (Syracuse “Economically Optimum Computer Software 
University) A. Sukerl (RADC) . Maintenance Strategies” 

• M. Horn. W. Thompson (Colum- “A Comparison of Results Obtained From Es- 

bia Research Corporation) tablished Software Reliability Measurement 

Models” 


2:45 p.m. BREAK 



3:00 p.m. PANEL NO. 4 


“MEASURING THE DEVELOPMENT PROCESS” 
DISCUSSANT: L Belady/IBM 


• S. Sheppard/E. Krucsi 
(General Electric) 

B. Curtis (ITT) 

• R. Cruickshank/J. Gaffney 
(IBM) 

• S. Moy (Logicon) 

4:30 p.ni. ADJOURN 


“Symbology and Spatial Arrangement: Effects 
on the Comprehension of Software 
Specific'ations” 

“Software Design Coupling and Strength 
Metrics” 

“A Tool for Software Design Evaluation” 
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J, Baizo, Computer Sciences Cor|x>ration 
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THE SOFTWARE ENGINEERING LABORATORY: 
An Approach To Measuring Software Technology 

F. McGarry 

NASA/Goddard Space Flight Center 


Over the past several years, we have seen the advent of newer and more disciplined approaches to 
the overall task of software development. Such approaches as structured design and development 
methodologies, improved management techniques, software metrics and measures, automated de- 
velopment tools, refined resource estimation and reliability models, as well as numerous other 
disciplines have been impacting the strategies of building, maintaining, and estimating the soft- 
ware process and product. 

Although the software development community has been presented with these numerous software 
development methodologies, each claiming seemingly to be more effective than the other, it has 
not been clearly understood, at least in the NASA/Goddard environment, what effects these tech- 
niques may have on different phases of the software development process. We have not really 
understood if structured programming, automated tools, organizational changes, resource models, 
or any of the other technologies would have any impact (either favorable or" unfavorable) on our 
own local software development process. It has also been quite apparent that it is not a trivial 
task to define what is a “better” software product. For these reasons, the Software Engineering 
Laboratory (SEL) was created at NASA/Goddard in conjunction with the Computer Sciences 
Department at The University of Maryland and with the Computer Sciences Corporation - the 
prime on/off-site support contractor for the subject environment at Goddard. The SEL set out 
to accomplish three basic tasks: 

(1) Provide a means of measuring and understanding the software development process at NASA/ 
Goddard. 

(2) .Measure the effects of available software development techniques as applied to applications 
programs at NASA. 

(3) Define, develop, and/or refine those methodologies, models, and tools which will have a 
favorable impact on the software process and product at N.\SA. 

The SEL was created in late 1976, and it spent most of the first full year defining factors and 
pertinent information that could reasonably and reliably be e.xtracted from software development 
tasks. It also defined an experimental process by which varying technologies could be applied to 
different projects so that the methodologies, models, and tools could be measured and evaluated. 
The experiments involved the application of different sets of software methodologies to different 
applications projects within the NASA/GSFC environment. 

In order to support the efforts of measuring effects of software techniques, it was necessary to 
define and implement an extensive monitoring process by which the details of all aspects of the 
software development process and product could be extracted for analysis. This involved the 
implementation of a software data collection mechanism by which the necessary detailed history 
of all aspects of a development project could be archived. 

The data collection process has involved five sources of information for the more than 30 projects 
that have been closely monitored since 1977. These sources include: 


F. McGarry 
NASA GSFC 
I of 15 



Data Collection Forms - which arc nilcd out by programmers, managers, technical support, 
etc. 

Automated Computer Accounting Information - such as number of runs, CPU time, etc. 
Automated Tools - such as code analyzers. 

Subjective Management Data - such as level to which certain approaches were followed. 

Personal Interviews - used as a follow up to the forms. 

These sources have provided data on about 35 development projects and have resulted in a data 
base of historical data approaching 45 megabytes of information. 

The projects that have been involved in the experiments of the SEL have ranged in size from 

2.000 lines of source to about 120,000 lines of source with most of the projects falling in the 

40.000 to 60,000 size range. All of the subject software has been developed to support flight 
dynamics requirements of various spacecraft missions supported by Goddard. None of the ex- 
perimental data were extracted from synthetic projects. 

Before committing resources toward the attempted measurement of the softw^are process and soft- 
ware product, the obvious question that arises is, “Why should we attempt to perform software 
measurements?'’ There are three reasons for such an effort: 

(1) To better understand the process in our particular environment. 

(2) To provide some rationale for setting standards or guidelines within a particular software de- 
velopment community. 

(3) To advance state-of-the-art technology in the overall software development process. 

To support these goals of the software measurement process, there are some basic requirements 
that must be fultllied in order to successfully arrive at any conclusions pertaining to the software 
being measured. First, we must attempt to understand approaches to existing software method- 
ologies and technologies so that we may clearly define just what we are trying to measure: i.e.. 
we must have access to some software technology base. Second, we must have access to a test 
environment so that the techniques of interest may be included in realistic experiments. We can- 
not assume that synthetic environments using synthetic projects would provide valid measures of 
any software technology being studied. The third requirement for supporting the measurement 
process is some training medium through which experimental subjects (whether they be managers 
or programmers) can be taught proper utilization of software techniques which are to be studied. 
The final requirement and certainly one of the most evasive and ill defined is a set of measures 
by which we can gauge both the software development process as well as the product. 

An attempt has been made in the SEL to provide the essential components needed in the meas- 
urement of software technology. Through 4 years of extracting information from development 
projects, the SEL has learned that the measurement process is not only difficult, but it is ex- 
pensive. In order to collect information, process the software data, and analyze them, an over- 
head as much as 25 percent has been enamntered with the projects under study. Although this 
overhead may seem extreme, it has found that there are immediate and subtle benefits from the 
overall measurement process which greatly reduce the actual total cost of the experimentation. 

F. McOarry 

NASAvGsic 
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Based on the SEL experiences, the importance of collecting software development data and meas- 
uring development methodologies cannot be overemphasized. Before we can ever expect to im- 
prove the process by which we develop software as well as the software product itself, we must 
understand our own environment as well as the possible effects that changing technologies may 
have on this environment. 
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THE VIEWGRAPH MATERIALS 
for the 

F. McGARRY PRESENTATION FOLLOW 
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AN APPROACH TO 

MEASURING SOFTWARE TECHNOLOGY 



NASA/SEL 








r.sFc 



MEASURING SOFTWARE TECHNOLOGY 

MAJOR REQUIREMENTS 
O 0 0 O 

TECHNOLOGY TEST TRAINING 
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FORM TECHNOLOGY BASE (TEAM) 

• UNIVERSITY OF MARYLAND 

• GSFC 
«CSC 


MEASURING SOFTWARE TECHNOLOGY 

THE SEL APPROACH 

lASE (TEAM) I ^ APPROACHES 
MARYLAND IDEAS 

MEASURES 


CREATE LABORATORY ENVIRONMENT [ 

• CSC & GSFC PROGRAMMERS 

• DEFINE CANDIDATE PROJECTS 

• PROVIDE DATA STORAGE/ANALYSIS 

MEDIUM 




CALIBRATE DEVELOPMENT PROCESS 

• EXTRACT DEVELOPMENT DATA 

• IMPACT APPLICATIONS TASKS 

(SCREENING TASKS) 


PERTURB NEW TASKS 

• APPLY SOME TRAINING 

• EXTRACT ADDITIONAL DATA 

(SEMI-CONTROLLED TASKS) 


PERTURB NEW TASKS - REFINE 
APPROACHES 




FORMS FOR DATA COLLECTION 
STORAGE SYSTEM 
PROCESSING PROCEDURES 


PROFILES OF SOFTWARE 

• STRENGTHS/WEAKNESSES OF 

EXPERIMENTS 

• REFINEMENT TO DATA PROCESSING 

• BASIC EVALUATION OF MODELS, 

METRICS 


1 • INITIAL MEASURES OF APPROACHES 

• MORE DETAILED PROFILES 

• MORE DETAILED EVALUATION OF 

MODELS, METRICS 


C=> 


NASA/SEL 














SAS-VCSFC 



MEASURING SOFTWARE IN THE SEL 

BASIS FOR ANALYSIS 


• LABORATORY EXPERIMENTS 20-25 PROJECTS 

• INFORMATION MONITORED I .3 MILLION LO.C. 

• PROGRAMMERS/MANAGERS REPRESENTED 75 PEOPLE 

• DATA EXTRACTED 40^ BYTES ON DATA BASE 

f^ORMS (15.000 FORMS) 

TOOLS 

SUBJECTIVE 

• METHODOLOGIES APPLIED 75 QUALIFYING PARAMETERS 

VARIOUS MODELS, 
TOOLS 

NASA/SEL 
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MEASURING SOFTWARE TECHNOLOGY 

SOME IIMDICATIOIMS FROM SEL EXPERIENCES 


• COST 

• GENERAL EXPERIENCES 
• PAYOFFS 



NASA/SEL 


MEASURING SOFTWARE TECHNOLOGY 

COST 

(AS % OF TASKS BEING MEASURED) 


SEL COULD 

EXPERIENCES BECOME 

• OVERHEAD TO TASKS (EXPERIMENTS) 5 - 10% K 04 

• FORMS ~ 

• MEETINGS 

• TRAINING 


• DATA PROCESSING 

• COLLECTING/VALIDATING FORMS 

• ORGANIZING DATA 

• ENCODING INFORMATION 


6 - 8 % 


• ANALYSIS OF INFORMATION—. 

• MEASURING METHODOLOGIES 

• DESIGNING EXPERIMENTS 

• DESIGNING ANALYSIS TOOLS 

• DEFINING MEASURES 


15 - 25% 10% 


• SUPPORT SOFTWARE—.. 

• DATA BASE- 

• CODE ANALYZERS 

• REPORT GENERATORS 


2 MY- THEN 1 MY/YR ’/z MY/Y 


NASA/SEL 


MEASURING SOFTWARE TECHNOLOGY 

DESIRABLE SUPPORT lIMFORMATION/DATA 


• PROJECT RESOURCE 

PROJECT LEVEL 

COMPONENT LEVEL 

• STATIC ANALYSIS 

• PERSONAL 

• CHANGE/ERROR 

• DETAILED COMPONENT 

• SUBJECTIVE PROJECT INC 

• PROJECT SUMMARY DESCRIPTION . 

• COMPUTER ACCOUNTING 

• COMPUTER USE 


SOURCE 


_(M) 

• PROGRAMMER 

_ (P) 

(P) 

_ (T) 

• MANAGER 

(M) 

.. IP) 

• TOOL 

- (P) 

IT) 

_ (P) 

• COMPUTER 


(C) 

(M) 


— % 1 W 1 / 

_(M) 


_ 1C) 


- (P> 
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EXPERIENCES FROM THE SEL 

INFORMATION FOR MEASURING 
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MEASURING SOFTWARE TECHNOLOGY 

SOME THOUGHTS FROM THE SEL 

• AVOID DROWNING IN DETAIL . . . "CANT SEE THE FOREST ..." 

9 TAKE TEST SUBJECTS INTO YOUR CONFIDENCE 

• SUBJECTIVE MANAGEMENT INFORMATION - VERY, V^ IMPORTANT 

• DONT DRAW CONCLUSIONS FROM SINGLE PROJECT 

• DONT BE OVERLY CONCERNED ABOUT "PSYCHOLOGICAL" FACTORS 

• REAL TIME FEEDBACK NEARLY IMPOSSIBLE, BUT . . . 

• AVOID EXTENSIVE- "WE NEED MORE DATA" 

"WE NEED MORE EXPERIMENTS 

"WE HAVE TO VALIDATE THE INFORMATION" 

• START WITH GENERALITIES - THEN REFINE 


nasa/sel 
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THE SOFTWARE ENGINEERING LABORATORY: 

Impacts of Experiments and Software Technology Changes in a Production Environment 

J. Page 

Computer Sciences Corporation 


THE VIEWGRAPH MATERIALS 
for the 

J. PAGE PRESENTATION FOLLOW 
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efFBCTS Of exnHIMBNTS « CHANGING TBCmOLOGY 
IN A nODUCT/ON BNVIWNMBNT 


Cm^GING TBCHNOLOGY 


BXPmHBNTS 




6JOC 


CHA-RACTSmTICS OP TMB 


TYPB SOPTyAVB 

&amific, GWUND MSSV 
f07iTRAN{SOMe ALC) 
mmGB SOK L.O. C- 
TYPICAL !8 MONTH DeveWPMENT 
LAPOB SCALE COMPUTETY USAGE 


ORGANIZATION 

SePAPATE ■SoPrUAPe SUPPOPT SeCT/OM 
TEAMS TOPMED U//7H/AI ONE SECTION 
^^OTHEPTOEPAPTMENTS PPOV/DE 
I analytical SUPPOPT 
CLOSE UOPKING PELAT/ON INTH 
CUSTOMEP (GSPC) 
OCCASIONALLY TEAMS INCLUDE GSfC 
PEPSOMNEL 


SOPTyAM BNVTRONMBNT 


\ 

i 


\ 

I 


MVBLO-PMSNT AfPliO AC H . j 

NO POPMAL STAND APDS ENPORCED j 

JNTEPNAL department APPROACHES UTILIIED 
SOME 3ATCH- SOME INTERACTIVE : 

NO STRICT CUSTOMEP STANDARDS 
HEAVY INTERACTION WITH CUSTOMER 
NO HEAVY USAGE OT DEVELOPMENT TOOLS 
COST ESTIMATES 3ASED ON HISTORY -e : 
'SMART' PEOPLE 

IN GENERA L - TOP DOWN DU! ED APPROACH 
TEAMS Of 4- iO PEOPLE 


mVBLOPMBNT TBAM 

SCieNTlPIC- COLLEGE BACKGROUND 
3- 4 YEARS EXPERIENCE 
GENERALLY NAVE WORKED ON ONE 
SIMILAR RROIECT 


AND CHANGING TBCHNOLOGY 
AmCT/NGTHB SOfWAB'B TSAMS 

vsTAaev'DATA'-peQumsDCmcoTiD Heep/m) popms k>pms mns 

• POPMAL TRAINING 

• ALT£R STANDAPD V£V£LOPP!BNT AJP^OACNSS 
• UTILIZATION OP N£W TOOLS R£QUIP£D 
• New/AZZ/r/ONAL TSAA1 JNT£RPAC£S 

• AmiTlONAL CONTRACT (task) 7?£QLUR£mNT3 
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uTiLiim AdVANCSd sonmn mvblotmsnt atproachbs (csc/ssC) 

SJBPS TAMBN 

• CANdlVATE TROireCTS IDSMTimH U/mM CSC 

« WRMAL TRAINING OT SPSCIFIC MeTHODOLOGY (TIMB CRITICAL) 

• PLAUSme SLTBSeT OP 'LSARNBD' APPROACH CeLBCTBD 
• MANAGBMBNT REINPORCBMBNT APRUBD 

• mTHODOLOGY AVcnJSTBV TO SLUT THB TROBLBM 
• APPROACHES SUSJSCriYBLY BVALUATBD 
® RETRAIN - AZ/Tl/ST - RETRY 
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JMPL6MBNTIN5 SOfTyAPB EXPiPIMBNYS AT CSC 

STBPS TAKBN 

• mfINB dcove Of XNfOfMATlON 'RBQU/'ReZ) 

• CRBAfEV SUPPOff 7ASHS (DATA PPOCBSSIND^ AMABTS/Sy . . •) 

• /A/ WBSB TPAimO 70 CANDWATA T7?0JBCTS 

(fOPMSy iWlONALBy 'PLANS ^ ...) 

% VBQLUPB DB7AILBD DATA DB PPOV/DBD fWM TASK 

♦ PBQUIPe USAGB Of SBLB07BD TOOLS 

• CAPPY OUT SC7?BfN/N0 BXPBT^/MBA/TS 

% TPA/N/A/& 

MOPB TASKS 1//TU APPPOACHBS ADJUSTBD 
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emCTS OP TUB changing TBCHNOLOGY 
IN THB CSC BNV/HONHBNT 


^ VRODucTivny 


• MLiA'diLiry 


A MBT/^ODOLO&y APPROACH -f- 5- fO% 

TH_i MSTHODOLOGy APPROACH H- io ^/o 
CORRECT PBOPLB STRUCTURB -i-i0-f5% 

Mf/HITBLY DOBS/y'r HURT 

TO BARLY To TBLL QUAArr/TAT/VB BPTBCTS 


• APPROACH TO SOPTWARB AdOPTm SBLBCT TBCHA/ZQUBS (ASBTAHDARI^ 

USA6B OP RBSOURCB MOCBL A7=^0ACHBS 
AS SUPPORT 

UTTiB BTTBCT fROHJ TOOCS 


• '&BNBRAC CRBATBD SOPTU/ARB BA/0. SCCT/O/Y^ 

LCAPNIU& CURVB SURPR/S/A/GLY ABSENT 
PtOPLB BBCOMB B/SCOURAGBD W/THOUT 

rb:/uporcbmbnt 

HA S HBPT CUSTOMBR HAPPY (6SPC ) 
shown HBBD Tor bducat/onac 
RBJNfORCBMBHT 

PROGRAMMERS RBLUCTANT TO 3L/NJ) 
ACCSPTANCe 


emcTs or expB-RiHeNTs 

tkoductivity mveo D/sc/Pim ro process 

’ !0 °/o OI/PPPSAD miMIZSD SY 3W£ S£A/eflT3 
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PSYCMiOG/CAl NO £V/d£NCe Of NAWTNOm £ffBCT 
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PANECY 300STS MORACE 

NAS CAUSED SOME DfSCONTEAlT 


APfROACNESro ElOEriYARE-—— PEOYmD AN AWARENESS OF SOFT-WARE 

TECNNOLOBY 

JDENTIFIED PLAUSIBLE METN 0 D 0 W 6 /E 3 
3 A 3 /S FOR &ENERAT/NB SOFTWARE GUIDELINES 


BfftCTS OB BXVBRmm AND CHANG/NO T£Cm!OLO&Y 

SUMMARY 


BxveR/MeNTs^ m not mv^: ro 3B ov^ly costly 

% S/ZOCfU) 3T (JSTX> TO Sl/PPOTT MOTHODOCO&Y SaSCT/OA/ 

% TTOQ'KAMMBTS TO UOT WANT :BL/ND ACCOPTANCN 
• 'TeiNTiOWBMONT - KSY TO ANY SOCCBSS 


% MBTL/OOOLOO/BS’ - 


TBT/N/7B ATVANTA&BS 

CANNOT XMTm/B TNT TWTUCT 


% MOOT is - st/ll VBTY UNCSTTA/N AS TO TATT/CL/LAN 
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THE SOFTWARE ENGINEERING LABORATORY: 
Measuring the Effects of Software Methodologies Within the Software 
Engineering Laboratory 

V. Basili and J. Bailey 
University of Maryland 


THE VIEWGRAPH MATERIALS 
for the 

V. BASILI PRESENTATION FOLLOW 
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WE HAVE EXAMINED A SET OF PROJECTS 

• DEALING WITH GROUND SUPPORT SOFTWARE 

• RANGING FROM 2K TO 101 K DEVELOPED SOURCE LINES 

• DURATION RANGING FROM 4.6 TO 17.4 MONTHS 

• EFFORT RANGING FROM 5 TO 138 STAFF MONTHS 

• AVERAGE STAFF SIZE FROM 1 TO 8 PEOPLE 

• PRODUCTIVITY FROM 413 TO 1068 DEVELOPED SOURCE 

- LINES/STAFF MONTH WITH AN AVERAGE OF 668 DEVELOPED 

- SOURCE LINES/STAFF MONTH 

• DATA COVERS DESIGN THROUGH ACCEPTANCE TEST 

• INCLUDES MANAGER. PROGRAMMER AND SUPPORT STAFF 



V. BasU 
C. of MaryUml 
2 of 8 



PROJECTS VARY WITH RESPECT TO THE SET OF SOFTWARE 
DEVELOPMENT TECHNIQUES USED AND THE EXTENT TO WHICH 

THEY WERE USED 

• THERE WAS FORMAL TRAINING FOR SOME PROJECTS 

EACH PROJECT WAS RATED WITH RESPECT TO 

• A LARGE SET OF FACTORS 

• environment, methodology. EXPERIENCE. PERFORM- 

• VALUES WERE GIVEN ON A SIX POINT SCALE 

• RATINGS WERE SUBJECTIVE 

• RELATIVE TO THE LOCAL ENVIRONMENT 

• DONE NEAR END OF PROJECT WITHOUT KNOWLEDGE OF THE 
PRODUCTIVITY RESULTS 

• BY NASA (McGARRY). CSC (PAGE) AND UNIVERSITY OF MARYLAND 
(BASILI) 


V. Baiili 
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RELATIONSHIP BETWEEN PRODUCTIVITY AND VARIOUS FACTORS 


• NO SIGNIFICANT RELATIONSHIP BETWEEN PRODUCTIVITY AND 
SIZE 

(NO POINT IN CATEGORIZING BY SIZE) 

• METHODOLOGY FACTORS 

, FOR ALL THAT SHOWED A DIFFERENCE AMONG PROJECTS. THE 
; CORRELATIONS BETWEEN METHODOLOGY AND PRODUCTIVITY: 


PDL 0.26 

FORMAL DESIGN REVIEW* 0.62 

DESIGN FORMALISM 0.38 

DESIGN DECISION NOTES* 0.62 

DESIGN WALK-THROUGH 0.28 

CODE WALK-THROUGH 0.19 

CODE READING* 0.58 

TOP DOWN DESIGN -0.13 

STRUCTURED CODE 0.02 

LIBRARIAN USE** 0.52 

CHIEF PROGRAMMER TEAM* 0.62 

FORMAL TEST PLANS** 0.51 

HEAVY MANAGEMENT INVOLVEMENT -0.09 

FORMAL TRAINING* 0.58 

TOP DOWN CODE 0.29 


*SIG. < 0.01 
**SIG. <0.05 
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OTHER FACTORS 


9 TRIED THE FOLLOWING: 

- CUSTOMER INTERFACE COMPLEXITY 
-icUSTOMER ORIGINATED PROGRAM DESIGN CHANGES 

- COMPLEXITY OF: APPLICATION PROCESSING, PROGRAM FLOW. 
INTERNAL COMMUNICATION. EXTERNAL COMMUNICATION. DATA 
BASE COMPLEXITY. JERRY'S GENERAL COMPLEXITY RATING 

- CONSTRAINTS: I/O CAPABILITY. TIMING, MAIN STORE 

- PROGRAMMING GROUP EXPERIENCE: MACHINE FAMILIARITY, 
LANGUAGE FAMILIARITY, APPLICATION EXPERIENCE, SAME TYPE 
BEFORE 

- HARDWARE CHANGES DURING DEVELOPMENT 

- PERCENT REAL TIME OR INTERACTIVE* 

- PERCENT PROGRAMMER INVOLVED IN SPECIFICATIONS 

• ALL BUT ONE SHOWED NO SIGNIFICANT CORRELATION WITH 
PRODUCTIVITY 

’SHOWED SIGNIFICANT DIFFERENCE AT 0.05 LEVEL IN WRONG DIRECTION 
II. E., HIGHER % REAL TIME -* HIGHER PRODUCTIVITY) 


V. 
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BASED UPON A SIMILAR STUDY BY DOUG BROOKS (IBM/FSD) 


• TRIED TO SEE IF METHODOLOGY HAD A SIGNIFICANT EFFECT ON 
PRODUCTIVITY 

• USED A STATISTICAL TEST TO SEE IF THE PROJECTS WITH HIGH 
METHODOLOGY USE CAME FROM A DIFFERENT ENVIRONMENT 
(WITH RESPECT TO PRODUCTIVITY) THAN THE PROJECTS WITH A 
LOW METHODOLOGY USE 

• THE DATA USED WAS BASED UPON A RELATIVE RANKING RATHER 
THAN AN ABSOLUTE RATING 

• THE APPROACH WAS 

- DIVIDE THE RATINGS FOR EACH TECHNIQUE INTO 3 CATEGORIES: 
LOW (-1), MEDIUM (0), HIGH (1) (DONE TO OFFSET DIFFERENCES IN 
SCALES) 

- ADD THE RATINGS TO GET A CUMULATIVE METHODOLOGY RATING 

- DIVIDE PROJECTS INTO GROUPS BASED UPON THEIR RATING AND 
ANALYZE USING THE MANN-WHITTNEY-U TEST (NONPAR AMETR 1C 
STATISTICS) 
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RESULTS 


GROUP: 

LOW 

MEDIUM 

HIGH 

RATINGS: 

(-11. -9. -9. -9) 

(2, 2. 2, 1. 0. -1. -3, -3) 

(12,11, 8, 5, 5,3) 

PRODUCTIVITY: 

535 DL/SM 

660 DL/SM 

768 DL/SM 

RESULT: 





LOW DIFFERENT FROM MEDIUM U HIGH (SIG. AT 0.05) 
HIGH DIFFERENT FROM MEDIUM U LOW (SIG. AT 0.03) 


GROUP: 

LOW 

HIGH 

RATINGS: 

(-11, -9, -9, -9, -3, -3, -1) 

(0, 1, 2, 2, 2, 3, 5, 5, 

PRODUCTIVITY: 

602 DL/SM 

710 DL/SM 

RESULT: 




LOW DIFFERENT FROM HIGH (SIG. AT C.05) 
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CONCLUSION 


WE CAN SHOW THAT METHODOLOGY HAS A SIGNIFICANT POSITIVE 
EFFECT ON PRODUCTIVITY. 


V. Casili 
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PANEL #2 


SOFTWARE COST/RESOURCE MODELING 

J. Golden/J. Mueller/B. Anselm, Xerox Corporation 
R. Tausworthe, Jet Propulsion Laboratory' 

R. Law ler, Boeing Aerospace Company 
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SOFTWARE COST/RESOURCE MODEUNG: 

John R. Golden, James R. Mueller, Barbara Anselm 
Xerox Corporation, Rochester, New York 


The purpose of this paper is to briefly review the methodology and model developed by Larry 
Putnam^ and present some results of their application at Xerox. 

In The Mythical Man-Month ,^ Brooks stated that time and effort are not linearly related. Put- 
nam’s software costing model, based on Norden^ Rayleigh product life c>xle concepts, enables 
managers to define time and effort quantitatively. The tradeoff laws governing attempts to re- 
duce development time by the adding of more people become highly non-linear, effort being 
proportional to the inverse fourth power of development time; and some development times are 
not possible at all. 

The key attributes of Putnam’s approach include the showing of trade-offs between time and 
effort, establishing a schedule (dynamic) for loading, establishing the feasible development range, 
a theoretical and empirical basis, an associated computational model, and the potential establish- 
ing of a baseline against which new technologies for software development can be measured. 


Methodology Summary (Detailed discussion in references I and 6.) 

Putnam used the ideas developed by Norden'* describing the project life beginning with the ob- 
servation that the rate of doing work (manpower) is equal to the work remaining times a linear 
function of time which Norden called the “pace” of the work (also called a linear “learning” 
curve by Parr^). The “pace” factor has dimensions of reciprocal time so in some sense corre- 
sponds to instantaneous natural frequencies of the project team. The equation states that the 
rate of accomplishment slows as the work remaining decreases and increases with time. That is, 
people tend to increase their “pace” more in response to the time remaining rather than the 
work remaining. 

If y (t) is the work done at time t, K is the total work to be done, and Xa is the development 
time, the differential equation is 

V = — = 2at(K - y) where a = (I) 

dt 

A particular solution of (1) is y = K, and the general solution of the reduced equation is 
y = c exp(-at2) ro that for y(0) = (h we have 

y = K{! -e-2i") (2) 

or in differential form for the power curve, 

V = 2atKe-"*' (3) 

A key property of Putnam’s approach was the development of the software equation which 
relates the principal parameters of development time, total effort, s\>iem size and the de- 
velopment environment. Putnam linearized (3) by using logarithmic riots of y t versus t- 
and noted that systems with similar levels of software w'ork or "diff.culty” lay along lines of 
constant intercept, K ly. Thus, he asserted that is a fundamental system parameter 

related to difficulty, that is, the software work to be done ‘not the 'ize of the system). 
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It is reasonable to expect productivity to be the same only for systems of the same difficulty. 
To determine the relationship of productivity and difficulty, Putnam took data from hun- 
dreds of systems and plotted productivity, P = S/K (where S is the number of source lines 
of code), versus difficulty, D = such that P = cD^. He found x to be approximately 

-2/3. Solving for S, we have 

S = (4) 

Equation (4) is called the software equation and c is a constant related to the technology 
level of the systems development environment. 

It is the software equation that indicates the non-linearity of Brook’s law: Effort is propor- 
tional to the inverse fourth power of development time. 

Finally, to examine the effect of time on difficulty, Putnam calculated the difficulty gradi- 
ent which consists of directional derivatives of D with respect to tjj and K. The magnitude 
of this gradient is dominated by the term in the negative development time direction (1^) 
and is approximately 2Klt^^ which tells us that difficulty increases as time decreases. The 
trace of constant gradient, K = bt|, where b is an empirical constant, when inserted into the 
software equation (4) yields the minimum development time: 

S = *cbi/3td^/3 (5) 

The constants b (there are five in Putnam’s current model corresponding to varying system 
complexity) determine the minimum development time for a given technology constant and 
system size by using (5). The corresponding K may then be calculated using (4). 

Thus, the only way to effect the minimum possible development time given a system of a 
certain size and complexity is by improving the development environment with new method- 
ologies and software tools and by adding more people! 


Results From Several Systems at Xerox 

A preliminary examination of the applicability of Putnam’s model to the Xerox environment has 
been conducted using Project Status Report data and results from interviews with the first line 
managers for four development projects (Figure 1). The estimates of time and effort predicted 
by the model for the minimum development were close to the actual figures for two of the four 
projects (A and B). The third project, C, exceeded the estimated minimum development time by 
20% and required approximately half the effort which the model would predict as required for 
the minimum time. This shows the inverse fourth power law of time versus effort since 1.2^ = 
2.07. The D project exceeded the minimum time by 15% to further illustrate the tradeoff. 

This evidence of the lime/effort tradeoff illustrates the potential savings attainable when develop- 
ment time is extended by a few months. 

Figure 2 shows the actual manloading for the B project compared with that estimated by the 
Putnam model. These results, similar to those experienced in working with the other projects, 
show that fie Norden- Rayleigh model basically describes the manloading pattern actually experi- 
enced by Xtrox. 

That allowing more time results in a lower cost may seem counterintuitive, for many managers 
see costs increasing -with increasing time. But these cases are associated with overruns usually 
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demonstrating poor estimating originally, or the refusal to effectively manage the inevitable 
changes required during development. Thus, when a longer time is planned , the result can be 
significantly lower cost. Conversely, if too many people are applied too st)on, the result will be 
higher cost with no positive effect on time. 

If. however, time is of the essence, the model will show tlie resulting cost and warn of potential 
problems as the minimum time is approached. Perhaps the most important feature of the model 
is the estimate of minimum development time. This gives us a tool to demonstrate the risk of 
Jrtifically imposed dates so prevalent in systems projects. Management can be appraised quanti- 
tatively of the inlierent risk of compressing schedules. 

The inqxirtance of breaking down a project into more manageable prices is clear, for not only 
will the system be cheaper, but early releases of those pieces will allow benefits to accrue earlier. 

If a project gets into difficulty, do not try to add more people, but consider descoping. If this 
is not feasible, accept the slip and learn from it. The Putnam model tells us that the applicafion 
of people to software development should follow the curve shown in Figure 2. or as “Brooks- 
second law” states; Adding more folks to a late project makes it^ later. 

It is ini|iortant to understand that this model is not a substitute lor project management but 
rather is a key input, namely the software component. Physical factors such as hardware acqui- 
sition and human factors such as training must be accounted for externally. 

One difficult and unresolved problem with the use of this or any similar model is system sizing, 
that is. extimating lines of code. It follows that a good requirements and system specification is 
needed. We believe the use of structured analysis and design will allow the appropriate correla- 
tions to be computed. For example, the number of process bubbles on a fully decomposed data 
How diagram and the number of data elements may replace lines of code as key inputs. 


SUMMARY: That a significant stride toward making software cost estimating and management 

“craft" rather than “witchcraft” is indicated in the following summary of our key points: 

1. Time and effort are not linearly related; effort is proportional to the inverse fourth power 
of development time. Therefore, time is very expensive. Improving the development en- 
vironment can offset this expense. 

2. Productivity (source lines per work-year) is not constant but is a function of system diffi- 
culty and the development environment. 

3. There is an intrinsic minimum development time which, given a system to be built of a eer- 
tain size and complexity, can only be affected by improving the development environment 
with new methodologies and software tools and not by adding more people. 

4 The imixirtance of implementing new methodologies is clear, tor with a good estimate ol 
size and complexity, the model gives predictive indications for time, effort and loading and 
therefore, can more readily accommodate change. 
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AGENDA 


• REVIEW OF PUTNAM'S SOFTWARE EQUATION AND IMPLICATIONS 

• SOME RESULTS AT XEROX 

• EFFECT CN DEVELOPMENT TIME ON RELATIVE ERROR 
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THE NORDEN-RAYLEIGH DIFFERENTIAL EQUATION 




RATE OF WORK ACCOMPLISHMENT 
IMG TIMES THE "PACE" 


IS EQUAL TO THE WORK REMAIN- 


LET I 

Y(t) = WORK COMPLETED AT TIME 
K = TOTAL WORK 

= DEVELOPMENT TIME 

THEN 


AND 


dY 

"dT 




ID 


Y(t) = K(1 - 


( 2 ) 


^ = JSl 

dt 


13 ) 
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LINEARIZE (3) BY TAKING LOGARITHMS AND PLOTTING 

dY/dt VERSUS 

log = log ^ - t2/2tj2 

K 

— REPRESENTS "DIFFICULTY" 

V 

A FUNDAMENTAL SYSTEM PARAMETER 
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PRODUCTIVITY (P = S/K) IS A FUNCTION OF DIFFICULTY, THAT IS, 
PRODUCTIVITY IS NOT A CONSTANT NOR CAN IT BE DETERMINED 
BY MANAGEMENT 

S /KX-* • 

P-cD-OR-.o.y 

PUTNAM FOUND x - 2/3 

SOLVING FOR S WE HAVE THE SOFTWARE EQUATION 

S = (4) 

OR 

K = 
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TIME AND EFFORT ARE NOT LINEARLY INTERCHANGEABLE 
BROOKS' FIRST LAW 

AND THE SOFTWARE EOUATION SHOWS US HOW NON-LINEAR IT IS. 
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WHAT IS THE EFFECT OF TIME ON DIFFICULTY? 


CALCULATE GRADIENT 


VD 




TERM IN NEGATIVE DEVELOPMENT TIME DIRECTION DOMINATES 
THE GRADIENT. 


SUBSTITUTING THE TRACE OF CONSTANT GRADIENT, K = BT^, 
WHERE B IS AN EMPIRICAL CONSTANT. INTO THE SOFTWARE 
EQUATION (4) YIELDS THE EOUATION FOR MINIMUM DEVELOP- 
MENT TIME; 


S = cB’/3tj7/3 


(5) 
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THUS THE ONLY WAY TO EFFECT THE MINIMUM POSSIBLE 
DEVELOPMENT TIME IS BY IMPROVING THE ENVIRONMENT AND NOT 
BY ADDING MORE PEOPLE. 


OR 

ADDING MORE FOLKS TO A LATE PROJECT MAKES IT LATER. 

BROOKS' SECOND LAW 
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Comparison of Resource Estimates 
Putnam vs, Xerox Actual 



DEVELOPMENT TIME - 
TAI FNDAR MONTHS 

. EFFORT-M.AN MONTHS 


PUTNAM MINIMUM 
TIME ESTIMATE 

ACTUAL 
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MINIMUM TIME 

.ACTUAL 
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PUTN.AM VALUE 
FOR .ACTUAL 
TIME 

A 

i:.o 

11.0 

124 

16< 

176 

B 

i:.7 

13 

83 

s: 

76 

C 

12.8 

15 

304 

150 

161 

D 

10.2 

11.7 

70 

51 

41 
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FIGURE 2 COMPARES ACTUAL LOADING WITH RAYLEIGH LOADING. 

THESE RESULTS ARE SIMILAR TO OTHERS IN XEROX SUGGESTING 
THAT THE RAYLEIGH LOADING IS CLOSE TO WHAT WE DO. 
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CONCLUSIONS 


• PLANNING A LONGER TIME RESULTS IN SIGNIFICANTLY LOWER COST 
(INVERSE FOURTH POWER LAW). 

• APPLYING TOO MANY PEOPLE TOO SOON RESULTS IN HIGHER COST - 
PERIOD! USE THE RAYLEIGH LOADING CURVE. 

• PAY ATTENTION TO THE MINIMUM DEVELOPMENT TIME. 

• BREAK PROJECTS INTO SMALLER PIECES. 

• USE THE TRADEOFF LAW IN PLANNING. 

• RECOGNIZE PRODUCTIVITY IS ?40T CONSTANT BUT IS A FUNCTION OF 
SYSTEM DIFFICULTY AND THE DEVELOPMENT ENVIRONMENT. 

• RECOGNIZE THE DYNAMIC NATURE OF SYSTEM BUILDING AND USE 
THE MODEL TO HELP MANAGE CHANGE. 
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RESULTS FROM SEVERAL SYSTEMS AT XEROX 


• PROJECTS A AND B ARE CLOSE TO MINIMUM TIME 

DpniFrT r EXCEEDS MINIMUM TIME BY 20% AND REQUIRED HALF THE 
rFroR?THl MlD“ WOULD PREDICT FOR MINIMUM. 


• TRADEOFF SEEMS "REAL" 


REFER TO FIGURE 1 
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• SPEND RESOURCES ON IMPROVING THE ENVIRONMENT TO HAVE 
REAL IMPACT. 

• TRY TO DE-SCOPE A PROJECT IN TROUBLE RATHER THAN ADDING 
MORE PEOPLE. (I AM NOT OPTIMISTIC ABOUT THIS PROCESS IF 
SYSTEM WAS NOT WELL DESIGNED.) 

• PUTNAM'S MODEL IS NOT A SUBSTITUTE FOR PROJECT MANAGE- 
MENT BUT RATHER IS A KEY INPUT. 

• SYSTEM SIZING POSES A MAJOR DIFFICULTY; LINES OF CODE IS 
FELT TO BE WANTING. 

• STRUCTURED ANALYSIS MAY BE AN ANSWER TO ENVIRONMENT 
AND SIZING. 
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EFFECT OF DEVELOPMENT TIME ON RELATIVE ERROR 

CALCULATE THE EFFECT OF THE DEVELOPMENT TIME, tj, ON THE 
RELATIVE ERROR, E = SI/S, USING PUTNAM'S SOFTWARE EQUATION: 

S = cK’/3tj4/3 

S IS TOTAL SOURCE LINES 

SI IS SOURCE LINES IN ERROR 

TO DO THIS WE CALCULATE THE TOTAL DERIVATIVE OF E WITH 
RESPECT TO tj, 
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S = cK’/3td«/3 

dE dE dS 

dtj dS dtj 

$1 dS 
■ S2 dtd 

■ -I 

- 

= - E cK’/3 -i td1/3/cKl/3tj</3 
o 



dE _ £ dtj 

E " ■ 

4 

log E = - y log tj + c, 

E = C 2 V /3 


SOFTWARE EQUATION 


SINCE E = SI/S 


(ES = S1) 


SEPARATING VARIABLE 
INTEGRATING 
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SOFTWARE COST/RESOURCE MODELING: 
Deep Space Network Software Cost Estimation Model 

Robert J. Tausworthe 
Jet Propulsion Laboratory 
4800 Oak Grove Drive 
Pasadena, CA 91103 


ABSTRACT 

This paper presents a parametric software cost estimation model prepared for JPL Deep Space 
Network (DSN) Data Systems implementation tasks. The resource estimation model incorporates 
principles and data from a number of existing models, such as those of the General Research 
Corp., Doty Associates, IBM (Walston-Felix), Rome Air Force Development Center, University of 
Maryland, and Rayleigh-Norden-Putnam. The model calibrates task magnitude and difficulty, 
development environment, and software technology effects through prompted responses to a set 
of approximately 50 questions. Parameters in the model are adjusted to fit JPL software life- 
cycle statistics. The estimation model output scales a standard DSN Work Breakdown Structure 
skeleton, which is then input to a PERT/CPM system, producing a detailed schedule and resource 
budget for the project being planned. 


•The research reported in this paper ^fcas performed at the Jet Propulsion Laboratory of the Catifornu Institute of Technology 
sponsored bv the National Aeroruutics and Space Administration, under contract NAS7-100. 
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1. INTRODUCTION 


The early-on estimation of required resources and schedule for the development and maintenance 
of software has been one of the least precise aspects of the software life cycle, and yet, an orderly 
and rational attempt is necessary to plan and organize an implementation effort. Such an ap- 
proach implies the need for a resource and schedule model that accepts as input the technical re- 
quirements to be achieved; the enormity of the task; the physical, environmental, human, and 
management constraints assumed or known to be in effect; the history base of similar and dissim- 
ilar experience; the means, alternatives, and technology available to the task; and a theory which 
is capable of correlating these parameters with measured results. 

The least precise of such models is one which relies entirely on experience, intuition, and luck. 

It is sometimes referred to as a “WAG”, or “Wildly Aspiring Guess.”* When a more formalized, 
mathematical model with some statistical verification can be formulated, the model appelation is 
upgraded to “Scientific WAG,” or “SWAG . 

The prediction of human programming behavior is a problem in estimating a series of events in a 
stochastic process governed by an unknown probability function. Tlie goal of a SWAG model is 
therefore to predict the events in such a way as to produce minimum variance (or risk). The 
optimum SWAG model can predict only to the limit imposed by the statistical distribution actu- 
ally characterizing the human activity. 

The optimal SWAG model would require the precise quantification of all technical, environmental, 
manaeement, and human behavioristic parameters, and would combine these into a mathematical 
formula producine maximum likelihood results. Lacking this precise quantification, the best that 
one may hope for in a SWAG model is that it accommodate the principal factors effecting the 
estimate variance (or risk) in a way that reduces the variance (or risk) from what it would be. 
had that factor not been included. 

Tliere are a number of SWAGs in existence. Fourteen software cost estimation models are sum- 
marized in Ref. 1, and nine were evaluated for use by the Jet Propulsion Laboratory in Ref. 2 . 
None of these, by itself, seemed to the author to contain sufficient range of application and 
adaptability to the diverse kinds of software being produced at the Jet Propulsion Laboratory, as 
to quantify the relevant cost factors and risks with sufficient accuracy to be useful. Taken all 
together, however, these models seemed to possess, in their union, the potential for as good a 
SWAG as could he obtained at the current state of the art. 

An IBM study (Walston-Felix. Ref. 3) reported the analysis of 50 software projects with respect 
to 68 variables believed to influence productivity. Of these, 29 showed a significant, high correla- 
tion with productivity, and were includi^d in their estimation model. 

A number of other models (General Research Corp., Doty Associates. TRW. Air Force Electronic 
Systems Division. Tecolote. Aerospace Corp.. et. al.. reported in Ref. 1, and the University ot 
Mary land. Ref. 4) provide productivity data with best-fit curves using many fewer parameters. 


•More oltcn, the is JitTcrcntly derived, but with the 


same eeneral uonnot Jtion. 
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Still other models, notably the Rayleigh-Norden-Putnam model (Ref. 5, 6) presuppose a few- 
parameter, specific mathematical model, which is then calibrated using available industry data to 
provide tradeoffs between effort, duration, and risk. 

Several models (e.g., Wolverton, Ref. 7) proceed to detail resource expenditures into the various 
phases of activity. This TRW model additionally compensates for task difficulty and some en- 
vironmental factors. 

Some of the models are fully automated, such as PRICE-S (Ref. 8), SLIM (Ref. 9), and SLICE 
(Ref. 10). The others appear to be calculative, or perhaps small programs, to be used by project 
planning staffs. 

The software cost model described in this paper is fully automated; it borrows and extends fea- 
tures from many of the models above. It utilizes 7 factors from the GRC model, 29 factors 
from, or similar to, the Walston-Felix model, and incorporates an inherited (or existing) code 
model due to the author. It utilizes the PERT estimation technique to estimate the expected 
size and variance of the software elements to be produced. It utilizes a modification of the 
Rayleigh-Norden-Putnam model to check on the confidence factors associated with the resultant 
resource estimates. It applies the estimated effort, staff, and duration to a standard Work Break- 
down Structure (WBS) developed for JPL Deep Space Network (DSN) software tasks, and auto- 
matically produces a task budget and schedule to be used at either the initial system/subsystem 
planning, software implementation planning, or software maintenance planning stages of a project. 

There are about 70 parameters within the model which relate to productivity, duration, staffing 
level, documentation, and computer resources. Another 70 parameters divide the total estimated 
effort among WBS subtasks: an additional 70 relate total duration to subtask durations. Sublask 
precedences, are preset (but adjustable), and arive a PERT/Critical-Path-Method scheduling 
algorithm. 

The outputs of the model include estimates and variance values for program size, staff productivity, 
effort, duration, staffing, documentation, and computer resources, together with confidence level 
checking, a complete scheduling early/late start/finish and float-time budget and a Gantt chart 
(schedule bar chart) of the planned activities. 

All parameters in the automated model are easily altered by a simple text editing process, without 
recompilation of the programs. For all its seeming complexity, the model itself is simple to use. 
Only a series of questions relating to size and environment need to be answered. 

The ensuing sections of this paper describe the model in more detail. The values of parameters 
used currently are subject to modification and refinement as further calibration and usage of the 
model proceeds. 

Concerning accuracy: at this writing, insufficient data has been collected from DSN projects to 
optimize the parameters of the model to fit DSN productivity, duration, etc. Therefore, the 
model accuracy is unknown, as pertaining to DSN prediction. However, the model does fit in- 
dustry statistics (or can be made to fit any of the cited source models) by proper parameter 
selection. For this reason, it is felt that the few' JPL data points that have been factored in will 
yield accuracy figures as good as; or perhaps belter than, the other models in their slated 
environments. 

The model is currently being used to estimate and plan all new programming activities involving 
DSN Data Systems implementations. 
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2. MODEL DESCRIPTION 

•nie Software Cost and Resource Estimation Model (the acronum SCAREM is not used') over- 
view appears in Figure 1 in data-flow-diagram format. Program size and staff productivity factors 
are separately estimated and then combined to estimate effort, staffing, duration, docurnemaficn 
and computer resources. The model produces uninflated dollar costs for documentation and Sm- 
puter resources. Both estimated mean and variance values for all resources are output. 

Estimated values are presented in the automated model to the user as advice. The user is admon- 
confide°nce%Lu^^^^ figures, adding sufficient margin to ensure project completion within a desired 

The model then accepts any two of the three parameters: effort, average staff, and duration 
These entries are checked against a model akin to the Rayleigh-Norden^utnam model, but al- 
tered to conform wnh power-law fits to measured data. A confidence factor is computed to the 
resources entered. The user may alter the input estimates, if desired, for another check. 

Once acceptable resources and duration have been decided, the model proceeds to produce a 
standard DSN software implementation work breakdown strueture and schedule without further 
input from the user. 


2.1 Estimation of Program Size 

software task is measured in “equivalent” Kilo-Source-Lines of Executable Code 
(KSLEC). A source line of executable code is defined basically as a source language statement 
occupying one physical line in the source medium that results in generation of object code 
reservation of storage, or definition of data type. Comments are excluded, as are statements 
merely defining labels and equivalences of identifiers. If several basic statements may appear on 
one physical source line, each such statement should be added separately into the KSLEC count. 

Source lines of new code are weighted differently than lines of reused code, in proportion to the 
relative amount of effort required to adapt the inherited code to the current task Even deleted 

contribute to the programming effort, and therefore increase the “eauivalent” 
KSLEC count, ^ 

The programming tasks involved w-h the generation of new code and reuse of existing code are 
depicted in Figure 2. The efforts to specify, produce, document, and test a new line of code is 
normalized to unity: the lines of code added, changed, deleted, and retested-onlv contribute to 
the equivalent line count according to relative effort. The extent of existing-mo'dule modification 
IS measured by the number ot lines added in, the number changed, and the number deleted The 
equivalent lines ot code produced is then defined to be a weighted linear sum of the lines of code 
in each category. 


The assumptions with respect to each component are the foIiowin»: 

(a) New' code is subjected to the entire standard implementation process. 

(b) The recognition of the reuse of existing cxide is made in the preliminarv design phase 
so code added, clianged or deleted from modules goes only tnrough subsequent phases. 

(c) .-\dded code takes the same ettort as new code in corresponding phases where activity 
takes place. 
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FIGURE 1. DSN SOFTWARE COST ESTIMATION MODEL DATA FLOWDIAGRAM 
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FIGURE 2 

SOFTWARE IMPLEMENTATION TASKS RELATED TO NEW AND INHERITED CODE ACTIVITIES 
INHERITED CODE 



(d) Changed code requires the same design and testing effort as new code, but less docu- 
mentation and coding effort. 

(e) Deleted lines from existing modules require reduced design, coding, and documentation 
effort, and no testing of the deleted lines. 

(0 Any module changed is completely retested and requalified. 

(c) Deleted modules require reduced architectural, interface, and detailed design considera- 
tions than new code; only that coding effort required to remove the unwanted code 
contributes to the coding time; no testing of the deleted code is possible; and documen- 
tation effort involves removal of entire sections of existing material and minor clean up. 
Retesting is covered in modules which interfaced with the deleted module. 

(W Retested unmodified code requires rcvalidation of the interface design and retestmg 
efforts only. 

Each of the size parameters is estimated by the PERT technique (Ref. 11) which produces a max- 
imum likelihood value and variance. 

The estimated value for the “equivalent lines of code” is composed of the weighted sum of the 
maximum likelihood estimate for each parameter, and its standard deviation is the weighted 
root-sum-square of the individual deviations. The weights are determined as the relative effort 
required for each category of code. 

2.2 Estimation of Productivity 

In this model, the productivity P is defined as total equivalent KSLEC (here denoted L) produced 
• divided by the total staff work effort (here denoted W), 

P = L/W. KSLEC/staff-month 

A number of data bases (e.g.. Refs. 5, 12, 13) have shown that L and W are well correlated 
through a power-law relationship 


W = LA'P 


wliere P[ is the average 1-KSLEC productivity rate. 


The vilue of Pi is set by primarily technology and environment. In tact, industry studies show 
tliat P, mav vary bv as much as 50: 1 (or more!) as a function of such factors. The value of a. 
however, in each environment where data is available, shows a relative constancy, at a value near 

iimty. 

ft seems intuitive, all other things being equal, that productivity cannot increase as the program 
M/e rises liowever. the le.ist-stluare-pov.er-law fits to data bases yield a-values ol 0. I (IBM). 
O'hif (Dots I. 0.086 (Univ. of MD). anu 0.075 (RADO. The consistency ol these ligures there- 
uire see. ns to indicate that utilization of tools and technology takes place on larger projects 
I where it counts) more than on smaller projects. 


fhe value for a currently being used is unity. 

^on>s.TV ati\ c- 


Thus, the model niav tend ,to be ^bme\vhat 
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Several models have eontributed to the formula by which P 
GRC (Ret. 14) and IBM (Ref. 3). The form of Pj is: 


I is calculated, principally those of 


P| - PqA 


where P^ is a constant factor, and A 
value of A is computed from factors 


IS a technology and environmental adjustment factor. The 
judged to be significant. 


A — AjAt . . . Aji 


where each factor \ ranges between a ma.ximum value and a minimum value for that factor de- 
pend.ng o.t the extent to winch the factor is present in the software task. The moo A to 
Amin currently adjusted to span a 50: 1 ratio of productivity. 


. — > Hstiination ot Inipleinentation Task Duration 

The IBM. University of Maryland, and RADC statistics suggest that the averase duration T r... 
ipured tor L KSLb.C and W staff-months effort is approximated by 

T = T,Wb 

where T, is tne 1-RSLFC average duration, and b is a time-factor exponent found from in.lnorv 
da a. and b - O.a.sb was the average power-law tor the more extensive IBM. RADC and GRC 


lA Average Si a IT 

The average staff, in persons, results from manipulation of the duration equation. 
S = W/T = (1/T,) W'-b 


The staffing exponent, I -b = 0.644, implied 
averaging 0.620 across industr\. 


in tl)e DSN model compares wit Ii measured values 


2.5 Documentation Sizing and Cost 

I-od’-'wb!^^^ nearly linear relationship between pages of documentation and lines of 

cidt. whereas the Uu\e''rsuy ot Maryland measured almost a s«iuare-roof relationship DSN -\- 
periemce over 6 Myk-lll Data System programs reu-aled an exponent about midwav in between 

40-i'nae-s o cri-r '■"•"on’m'-'nded that documentation be aln.u 

4^?0 pages per KSLLC tor programs m the 30 kSLI-.C vicinity. The formula used in the model 
tor Ihc number ot pages ot documentation is miuuu 


D = DjL^ 


The model currently uses Dj 


^^0 and d 


0.S.> to match the DSN experience and euidclincN. 


The documentation cost is found 
m the current model. 


by a straight dollar-per-p.ige jate; a tigureof S.'U page is iisci 


K. lausiftiirthe 
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2.6 Computer Resources 


IBM and TRW give statistical figures for computer time costs as functions of lines of code and 
total effort, and also as a fraction of total cost. The DSN, however, mostly has dedicated mini- 
computers for which operational costs are not assessed to the implementation task on a usage 
basis. TRW does, however, also estimate a linear relationship between CPU time required per 
machine code instniction, of about 25.2 CPU hours per thousand instructions. 

The DSN model compuler CPU resources as 

C = CiL"' 

The exponent value c = given by Walston and Felix (who give dollar costs, rather than CPU 
time), is adopted to account for the general trend of CPU time with program size. 

If CPU dollar cost is relevant, the model computes this at a straight dollar-per-CPU-hour figure 
(zero in the DSN model, but a parameter is available for other applications). 


2.7 Confidence Level Computation 

The values predicted by the SWAG model are average values based on statistics taken over many 
projects. Tile estimated values represent but one set of resources that, on the average, produce 
the intended success. However, effort and time can be traded (to some extent) to produce other 
equally valid project scenarios. 

The Rayleigh-Norden-Putnam model has a time-effort relationship for checking the reasonability 
of resource values other than the average values produced by the SWAG. However, in order to 
use this model, several adaptations were felt to be indicated. 

Fitst, the basic differential equ,.tion was modified to accommodate a nonlinear factor (or 

learning curve). The model assumes tlie work equation to be of the form 

w' = .AtMK - w) 

in which w - w(t) is the cumulative work effort up to time t. R is the total life-cycle effort, and 
r is the "pace-of-work" exponent. 

Second, the model assumes the following parametric form of the software equation: 

L = c.,wPtM 

The factor f = q p sets the lime-effort tradLvff law. Putnam uses p = L3. q = 4 '3 (f = 4), and 
such a value may well be valid for large projects in his data base. However, for the smaller 
projects typifying the DSN, an f factor which would require only 1.5 times the effort (3 times 
the staff) to reduce schedule by a factor of 2 seems to be more within the DSN experience 
(more data is needed here). 

Based on these moditlcations. it is possible to solve for p. q, and r, and c^ in terms of the paranr 
elers of observed average |X)wer-law relationships between L, W, and t. 

f = 0.5S5 q = 0.4S4- Cp = 0.062 L.-\ 

p = 0.S2S r = LSI 

R. Tausvkurthtf 
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This model is used to compute confidence for any values of L, W, and T proposed for the project. 
The software equation is used to estimate the margin over or under the average project figures. 

By assuming that the statistics are log-normal (verified by the RADC data base), the confidence 
factor in producing L KSLEC in W staff-months effort and T months duration is 

Conf=p{La,n^L,w<W,t<Tj 

Computation of the confedence level involves finding the optimum operating point on the 
software-equation curve for margin calculation, and then numeric integration of the normal 
probability function. 

One interesting relavation to the author was that the probability of success in not expending more 
than W staff-months of effort and not requiring more T months duration, for the average SWAG 
estimate case, is only about 25^c. Moreover, a significant amount of bias in W and T is required 
to raise the confidence to 50%. 

Management should not despair, however. What the confidence limit indicates for average projects 
is that one out of four will go OK; the others will require some form of management intervention, 
in the form of schedule or resource extension. 


3. TYPIC.AL OPERATION OF THE MODEL 

Blank forms and/or CRT prompted inputs are used to specify the parameters needed by the cost 
model prosram. Outputs are selectable, and include a WBS task file, PERT plan, and schedule. 
The WBS task file can be edited to modify task titles, precedences, allocated resources, or dura- 
tions in addition, new tasks may be entered and actual completion or need dates may be affixed, 
so that the PERT and schedule portions of the program form a project detailed planning and 
control tool. 

Typical inputs and outputs arc shown in Appendix A. 


4. SUMMARY AND CONCLUSION 

The Software Cost Model reported here is the first of a series of planned refinements. As the 
model is used and as performance data are coUected, no doubt changes will be made; adjustments 
of parameters, alteration of formulas, modifications of formats, new input data types, and addi- 
tional kinds of outputs. E.xtensions currently envisioned are the automated transfer of the WBS 
data-base generated by the model into the project control system currently bemg used in DSN 
Data Systems implementation projects, and the refinement of the model to include non-linear 
scaling of overall effort and duration into individual task requirements. 

If the model even now. seems complex, then it is justly so, for the factors which affect human 
performance are eenerallv complex and unpredictable, except in statistical terms, One sample 
function chosen from a stochastic ensemble is hardly ever “average” or “typical”. One must ex- 
pect variations between actual behavior and predictions made by any model. 

The directions for the future are to refine the model for greater accuracv- (within human perform- 
ance estimation capacitv limits), to extend the utility of the model throughout the entire life 
cycle, and to provide the basis for indicating where, and in what form, new software technology 
is needed. 
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appendix a 

EXAMPLE OF OPERATION 


This Appendix contains a sample of thejquence of inputs and outputs 
Network Software Cost Estimation Model. 
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TITLE: VERSION CONTROL EDITOR 

ECR/ECO: eSO.176 

S’JBSYS: X21.6 


CDE: Angus Day 

PROG. ID.: HLT-D2-OP-D.2 

Date Estiirated; J4NOV80 

Model Data Version 1.3 310CT80 


Answer the following items to the best of your estiitation. 


level)? 

3.5 


3.3 

level) ? 

3.1 

level) ? 

6.9 


6.6 

level) ? 

6.3 

level) ? 

.4 


.3 

level) ? 

.2 

level) ? 

.7 


.6 

level)? 

4 

• n 


2. How much code exists in modules requiring modification? 

Maximum value, kilo-linec executable source (99% conficence 
Expected value, kilo-lines executable source? 

Minimum value, kilo-lines executable source(99% confidence 

3. Hov much code will be deleted from these existing modules? 

Maximum value, kilo-linos executable source(99« confidence 
Expected value, kilo-lines executable source? 

Minimum value, kilo-lines executable source(99‘ confidence 

4. How much code will be added to these existing modules? 

Maximum Vw^lue, kilo-lines executable source (99k conficence 
Expected value, kilo-lines executable source? 

Minimum value, kilo-lines executable source(99% confidence 

5. How i*uch code will be changed in other ways in these modules? 
Maximum value, kilo-lines executable source (99* confidence level)? 
Expected value, kilo-lines executable source? 

Minimum value, kilo-lines executable source (99% confidence level)? 

6. How much code will be deleted as entice modules from existing code? 


7. How much of the remaining existing code must be retested? 
Maximum value, kilo-lines executable source (99i confidence 
Expected value, kilo-lines executable source? 


(C-90, 91-99, 100)? 

How many different kinds of input/output data iters per 1000 ’^nes of 
new or modified code(>6C, 16-30, 0-15)? jg. 


1 . 


level) ? 

1.4 

level) ? 

1.3 

■ . 1.1 

level) ? 

2.1 

level) ? 

1.9 

1.5 

red 

91-99 


9. 


10. Overall complexity of program and data base architecture 
(high, medium, lew)? 

11. Complexity of code logical desicnfhigh, medium, lo*-)? 

12. What percent of the programming task is in Assembly language? 

13. What percent cf the new or modified code must be storage-optimized? 


MEDIUM 

LOW 

9 

9 
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I K> 


14. 

15. 

16. 

17. 

18. 

19. 

20 . 
21 . 


22 . 


23. 


24, 


25. 


26. 


27. 

28. 

29. 

30. 

31. 


32. 


33. 

34. 

35. 


What percent of the new or modified code must be timing-optimized? 9 

What percent of the total programming task is 'easy'? 20 

What percent of the total programming task is 'hard*? 30 

When is work to start, on the (FKD/FDD, SRD, SDD) ? FRD/FDD 

What percent of the total program requirments will be established 

and stable before design, and will not be altered before delivery? 80 

What percent of the requirements are likely to change slightly before 
delivery, but will do so under baseline change control? 10 

What percent of the requirements are likely to change more drastically 
before delivery, but will do so under baseline control? 5 

Complexity of program functional requirements (high, medium, low)? LOW 


Expected user involvement in requirements definition 
(much, some, none)? 

Customer experience in application area (much, none, some)? 

Customer/implementor organizational interface complexity 
(high, normal, low)? 

Interfaces with other SW development projects or organizations 
(many, few, none)? 


MUCH 


SOME 


NORMAL 

FEW 


Efficiency of implementing organization (poor , ok, good)? 

Overall implementation personnel qualifications and motivation 
(low, average, high)? 

Percentage of programmers doing functional design who will also 
be doing development ;<?5 , 25-5C, >50)? , . 


GOOD 

HIGH 

25-50 


Previous programmer experience with application of similar or greater 
size and complexity (minimal , average, extensive)? AVERAGE 


What is the average staff experience, in years, obtained from work 
similar to that required in the task being estimated? 6 

Previous experience with operational computer to be used 

(minimal, average, extensive)? MINIMAL 

Previous experience with programming language (s) to be used 

(nini'^'al, average, extensive)? MINIMAL 


Vse of top-down methodology ( lew, medium, high)? 


HIGH 


Lse of structured programmer team concepts ( low, medium, high)? HIGH 

Tsr of Structured Ptocramming ( low, medium, high)? KIG!J 
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36. Use of design and code inspections(low, QA, peer)? OA 

37. Classified security environment for computer (yes, , no)? NO 

38. Hardware under concurrent development (much, some, none)? NONE 

39. Percent of work done at primary development site 
(<70, 70-90, >90)? 

40. Development computer access node (remote, scheduled, demand)? DEMAND 

41. Percent of development computer access availability ( <30, 30-60, >60)? 30-60 

42. Quality of SW development tools and environment (poor , ok, good)? OK 

43. Maturity of system and support software (buggy , ok, good)? OK 

44. Overall adverse constraints on program design 

(severe, average, minimal)? MINIMAL 

45. Is the program real-time, multi-task (chief ly , some, no)? NO 

46. to be adaptable to multiple computer configurations or environments 
(yes, , no)? 

47. Adaptation required to change from development to operational 

t nvi r onment (much , some, minimal)? MINIMAL 


R. Taus%»orthe 
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estimated Overall Pat 2 uneters: 


'average value 


+l-sigma 

Adjusted Lines of code- 6182 SLEC 
6280 60B5 

Effort- 26.5 person-ronths 
453 15.3 

Sta^f productivity- 233 SLEC/staf f-nonth 
404 135 

Duration- 15.7 months 
19.0 12.9 

Avg. Staff- 1.7 

2.S 1*0 

Docur-e jtion- 537 pages $16. IK 
645 $19. 4K 

CoiTDuter CPU time- 319 hours 

476 212 $Ci.0K 


$13. 4K 
$0.0K 


Use these figures to arrive 
requirements. Include factor 
and confidence levels. 


at Effort, Duration, and Staffing 
s to provide acceptible risk 


-1-signa 


Values specified are: 

Kilo-lines of code- 32.0 
Effort (person-months): 26^0 
Duration (months): 2,0 


Average staff (persons): 


For the nur-hets you have enteted, a reasonableness chect ndicates hat 
the average oro:cct iould produce 7303 lines of code, using 32 sta---^ohs 
of resources' and 16 nonths of duration, with an average of 2 »,er--ns, 
for a productivity of 228 SbEC/staf f-r.ontn. 


The level of confidence in delivering 6182 lines of code 
on-tine and within tesources= 33 

Is output to be saved in a file? 

Name of output file to be created: 

Schedule start date; 


Y 

VCEDIT 

17NOV80 


Select desired outputs and output media 
defaults. Defaults are lA, 2A, and 3A. 


, or enter RETURN on 
Choices are: 


ly 


for 


l«Cantt Chart A*file 

2- PERT data, 132 width B-line printer 

3- PERT data, 80 width 


Choice (s) : 


13,23,3A 
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! TITLE: VERSION CONTROL EDITOR 

! ECH/ECO: e30.:76 
; SVbSVS: X21.6 


PAGE 1 

— ! 

COE: Angus Day ! 

PROG. ID.: HUP-D2-OP-D.2 I 

ST ATI'S /xS OF; 14 NOV 80 I 


1 

! CODE 

• 0 . 

*1- 
• 1.1 
* 1.2 

! 1 . 2.1 

* 1 . 2.2 

• 1.3 

• 1.4 

• 1.5 

I 1.5.1 

• 1.5.2 

• 1.6 
• 2 . 

• 2.1 
• 2.2 
! 2 . 2.1 
« 2 . 2.2 

* 2*3*’ 
*3. 

* 3.1 

• 3.2 

1 3.2.1 

• 3.2.2 
I 3-3** 

* 3.4 
•4. 

I 4.1 
! 4.1.1 

• 4.1.2 

I 4!i!3 
: 4.1.4 

! 4.1.5 

! 4.1.6 

I 4.2 

• 4.2.1 

! 4.2.2 

4.2.3 

• 4.3 

* 4.4 

* 4.4.1 

• 4.4.2 

* 4.4.3 


TASK 


ST/VRT 

i*y£ Plans, Reqts, & Design 
Define Subsys Reqts 
fLD 

Nrite all sections 
Edit and release FRO 
Level B Review 
Define Sys Architecture 
FDD 

Write all sections 
Edit and release 
Level C Review 
SW Planning and Reqts 
Define Software Reqts 
SRD 

Write all sections 
Edit and release 
Level D Review 
Sv; Architecture and Design 
Define S\; architecture 
SDD 

Write all sections 
Edit and release 
System Interface Design 
Level E Review 
Sv; Detailed Design & Prod 
SSD 

Write Sections 1,2,3 
Write Section 4 
Write Section 5 
Write Section 0 
Write Section 7 
Edit and release 
scv. 

Write preliminary draft 
Cc’^plete all sections 
Edit and release 
Hich-levfcl Design Review 
Xor.ule Production U Inteq 
Executive and control 
I/O y.odules 
Interface handlers 


EFF : 

E-START 


L-FINSH ; 

FLT! 

0 ; 

17NOV80 


17NOV80 

01 

0 ; 

9JA».‘81 


9JA.N81 

01 

15 : 

17NUVB0 


3DEC80 

01 

0 : 

fcDECdO 


8DEC80 

Oi 

4 : 

17!.OVbO 


5DEC8C 

9! 

2 ; 

SDECeO 


8DEC8C 

0! 

3 : 

eDLCfaO 


lODECQC 

0! 

lb : 

1CDEC80 


31DEC80 

01 

0 ; 

7JANU1 


7JA.N81 

0! 

4 : 

1CLEC80 


6JA.M81 

12 1 

2 : 

6JANB1 


7JA.N81 

0! 

3 ; 

VJAfiel 


9J/uN81 

o; 

0 : 

'UFElibl 


16FEDB1 

O: 

25 : 

9JANB1 


4FE1381 

Cl 

C ; 

12: Llifil 


12 FEB 01 

Oi 

4 : 

91AriKl 


llFEBbl 

21! 

2 : 

llELDbl 


12FEU81 

0! 

2 : 

12FLLibl 


16FEB81 

Oi 

0 : 

eAFKBl 


6APR91 

0! 

31 : 

16r*tliSl 


19KAR81 

0! 

0 : 

2 APR 81 


2APR81 

0! 

4 : 

16FLH81 


1 APR 81 

30! 

2 : 

1APH81 


2APR81 

01 

22 : 

16r LUbl 


1APR81 

20! 

2 : 

2 APR 81 


6APU81 

0! 

0 : 

22DECbl 


22DEC81 

0’ 

0 : 

23FED82 


11KARB2 

12 1 

6 : 

6 APR 81 


IflDECai 

1751 

18 : 

eAPRBl 


24APR81 

o: 

43 ; 

24APKU1 


IbDECBl 

142 1 

4 : 

CAPHbl 


18DEC01 

176’ 

9 : 

f-.MRtll 


18DEC81 

1731 

6 ; 

lt:rUUi2 


11VAH82 

12 ; 

0 ; 

21FK!»ii2 


11NAU02 

13 ! 

8 : 

2 4 APR H] 


4yAYSl 

o; 

9 : 

2 JL.‘!i:-;l 


18DEC81 

1331 

4 ; 

1 i iui2 


21 MAR 8 2 

13: 

2 : 

2i-rAVbi 


2JDNbl 

0! 

0 : 

'ibELCei 


lUDSCei 

c: 

lb : 

^rAYbl 


29VAY81 

O' 

13 : 

2:VN31 


26JUN81 

0. 

le : 

2CJl’Nbl 


23JL'L81 

o; 


WHO 
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PAGE 2 


I 

I 

I 

» 

I 

( 


TITLE; VERSION CONTROL EDITOR 
ECR/ECO; e80.176 
SUDSYS: X21.6 

CODE : 


WHO 


• 4,4,4 

• 4.4.5 

• 4.4.6 

• 4.4.7 

• 4.4.3 

• 1 ,^ 
i 4.5 

! 4.5.1 

; 4.5.2 

‘ 4.6 
*5. 

• 5.1 
1 5.2 
! 5.3 

! 5.3.1 

; 5.3.2 

• 5.4 

• 5.5 

• 5.6 

! 6 . 

' 6.1 

• 6.2 

• 6.3 

• 6.4 

• 6.5 

! 6.6 


Function A 
Function B 
Function C 
Function D 
Function E 
Function F 
Special Tauks 
Support software 
Otner 

Acceptance Readiness Rvw 
SW Test and Transfer 
Verification tests 
Contingency 

Write all sections 
Edit and release 
Acceptance tests 
Der.onstr aticn tests 
"'ransfer. CDL to COE 
Mat Tas<s and Milestones 
CDE Activities 
Develop prelm budget 
Develop Sys Irpl Plan 
Draft Software Inpl Plan 
Revise Iirpl Plan 
Q,\ Audit 


•FINISH 


1 


CDE: Angus Day I 

PROG. ID.; HUP-D2-OP-D.2 I 

STATUS AS OF: 14NOVBO I 


EFF 

E-START 

; L-FINSH : 

FLTI 

18 

23JUL91 

; 17 AUG 81 

0! 

18 

17AUGbl 

; 11SEP81 

01 

17 

IISEPLU 

; eocrei 

01 

17 

GOCTal 

: 20OCT81 

ot 

17 

290CT81 

; 23NOV01 

Oi 

17 

23NOVU1 

; 18DEC01 

o; 

0 

lOJPNUl 

; IbDECei 

132 5 

12 

2JUNbl 

; 18DEC81 

132! 

6 

2JUNdl 

; 18DEC81 

135! 

2 

IBDtCHl 

; 22DEC81 

0! 

0 

1HV.AHH2 

: 18M/J182 

01 

28 

22DKC81 

t 1FEB82 

01 

25 

9APR81 

: 11MAR82 

2ie; 

0 

19FEb82 

• 18MAR82 

191 

14 

2JUN81 

: 1FED82 

1591 

2 

l8FEBb2 

: 11MARB2 

14; 

20 

1FEB&2 

: 18FED&2 

Ci 

22 

lbFElib2 

: 11KAI^82 

01 

7 

llMAJ<e2 

: 18MAK82 

Oi 

0 

loDUCdO 

; 18 MAH 8 2 

3131 

37 

17NOVb0 

; 18MAH02 

3131 

3 

3DEC80 

; 5DEC80 

0! 

3 

31DEC80 

: 6 JAN 81 

Cl 

6 

4FEB81 

; llFEBBl 

01 

12 

19MARB1 

: 3APR81 

Ci 

26 

lbFEHB2 

: ^ AMAH 82 

6! 

0 

ieNAKB2 

; lbVAHB2 

0! 
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task io5°DEC MKR APR MAY JUH JUL_AOG_SEP_^_HOV_DEC_J^^^ 

T + 


» 4.4.4 

» 4.4.5 

» 4.4.6 

• 4.4.7 

• 4.4.8 

• 4.4.9 
I 4.5 

I 4.5.1 
I 4.5.2 

• 4.6 
•5. 

• 5.1 
I 5.2 
I 5.3 

I 5.3.1 
I 5.3.2 

• 5.4 

• 5.5 

• 5.6 
16. 

I (.1 

• (.2 

• 6.3 

• 6.4 

• 6.5 
I 6.6 
•FINISH 

I 


Function A 
Function D 
Function C 
Function D 
Function E 
Function F 
Special Tasks 
Support software 
Other 

Acceptance Readlneaa Rvw 
SW Test and Transfer 
Verification teats 
Contingency 
STT 

Write all scctlona 
Edit and release 
Acceptance teats 
Demonstration testa 
Transfer# CDE to COE 
Kgt Tasks and Mileatonea 
CDE Activities 
Develop prelim budget 
Develop Sya Impl Plan 
Draft Software Irapl Plan 
Revise Impl Plan 
QA Audit 


— 
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. A“»»v . 


. • 




•A* 


a a a a 

mmmm 


• Sv • • 


. • 


• 


• 



• 

. • 

. A»»V . 


• • 

. • 

« « 

. • 

. • 

e 

e 

• 

• 

e 

• 

« 

. . • 

# • • 

. • • 

• 

• 

• 

• 

• 

. • 

• • 

. • 

■•A . • 

. ••■A • 

• .*A . 


. • 

. A***** 


mmmm 

mmmm 






■ 

• 

• 

• 

1 

t 

i 

■ 

• 
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A 


. 



• « t 

• 


« • 

• • * 


• -A 

, , 

• 


• 

. . • 

• 


. • 




-A . 

, 


e 

. • • 



. • 



. 


•■A 


• 

. . • 

• 


e • 

• • 

« aS-AV • 




• 



. • • 



• e 

. . A . 


HANIFINISH 

DYSIDATE 


18I17AUG81 
leillSEPBl 
171 60CT81 
17I290CTB1 
17)23N0V81 
17118DEC81 
0I18DEC81 
1211BDEC81 
6118DEC61 
2I22DECB1 
0I18HAR82 
281 1FEB82 
25111HAR82 
0I18HAR82 
141 1FEBB2 
2tllHARB2 
20|18FbBB2 
22111NAR82 
7I18HAR82 
0118HAR82 
37I18HAH82 
31 5DEC80 
31 6JAN81 
6I11FEBB1 
121 1APR81 
26I18HAR82 
0118HAR82 


LEGEND! 


•-ACTIVITY 
S-LATE START 
<-BVENTS PAST 


A-EARLY FINISH 
V-LATE FINISH 
V-ACTUAL FINISH 


•-CRITICAL PATH ITEM 
NOTE! ABSENCE OF $# 8, A# OR V INDICATES MERGED DATES 
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300CT80 


PAGE 3 


TITLE I VERSION CONTROL EDITOR 
ECR/ECOt e80.176 
SUbSYSi X21.6 


+- 

CDEi Angus *)ay 

PROG. ID.: HUP-D -OP-D.2 

STATUS AS OF: 14NOV0O 


• 1 . 

• 2 . 

•3. 

•4. 

‘5, 

16. 

I- 


1 

TASK 


Sys Plans, Reqts, L Design 
SW Planning and Reqts 
SW Architecture and Design 
SW Detailed Design 6 Prod 
SW Test and Transfer 
Ngt Tasks and Hilestonrs 


J9S0 1981 1902 I MANIFINISH 

NOV DEC JAN FEB MAR APR HAV JUN JUL AUG SEP OCT NOV DEC JAN FEB MAH APR I DYSlDATE 



« • . . . 

, , ■■••■A . • . • 

. . • ■■■••■A • • 


. . 




mmmmmmmSmmmummmmmmS 

...r«........«« — AV 


I 511 9JAN81 
I 33|16PEBei 
t 611 6APRB1 
I 287I11MAR82 
I 118I1BHAR82 
I 87I1BMAR82 
1 


LEGEND: •-ACTIVITY 

S-LATE START 
<-EVENTS PAST 

NOTE: ABSENCE 


A-EARLY FINISH 
v-LATE FINISH 
V-ACTUAL FINISH 

$, S, A, OR V INDICATES 


•-CRITICAL PATH ITEM 
MERGED DATES 


^ ' 

o 



run VIHWGRAPH MATERIALS 
for the 

R. TALSWORTHE PRESENTATION FOLLOW 
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DEEP-SPACE NETWORK 
SOFTWARE COST AND 
RESOURCE ESTIMATION MODEL 

PROTOTYPE 

Robert C. Tausworthe 


\ 


DSN SOFTWARE COST ESTIMATION MODEL 


• COMBINES AND REFINES A NUMBER OF MODELS 

• USES ABOUT 60 SIZING AND PRODUCTIVITY FACTORS TO PREDICT 

RESOURCES. DURATION, STAFFING. COMPUTER CPU, AND 
DOCUMENTATION REQUIREMENTS 

• COMPUTES CONFIDENCE FACTOR FOR DELIVERING ON-TIME AND 

WITHIN BUDGET FOR ANY PROPOSED EFFORT AND DURATION 

• PRODUCES SCHEDULE AND RESOURCE ESTIMATES FOR ABOUT 50 

INDIVI DUAL TASKS MAKING UP A PROJECT 

• IS SPECIALIZED TO OUTPUT A STANDARD WORK BREAKDOWN 

STRUCTURE (WBS) FOR DSN DATA SYSTEMS TASKS 

• PERMITS ALL MODEL PARAMETERS TO BE CHANGED BY SIMPLE 

EDITING PROCESS 


R. Taus«t>rthe 
NASAJPL 
27 of 46 





PARAMETER VALUES 
PRINCIPAL SOURCES 


• IBM DATA (WALSTON-FELIX) 

• RADC DATA BASE (NELSON) 

• UNIVERSITY OF MARYLAND (FREBERGER-BASILI) 

• GENERAL RESEARCH CORP(CARRIERE-THIBODEAU) 

• TRW (WOLVERTON) 

• QUANTITATIVE SOFTWARE MANAGEMENT. INC. (PUTNAM) 

• JPL WBS ARCHIVE DATA 
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BtCIN N"* 
EXECUTION^ 


I 



SOflWARE 
AND TASK 
CHAKACIERISIICS 


) 




DATA FLOW DIAGRAM 



i . 
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CODE SIZING MODEL 


INHERITED CODE 


UNCHANGED, UNRETESTED 
CODE . 


NEW MODULES 


MODULES 

REMOVED 


MODULES 

MODIFIED 

• ADDED CODE 

• DELETED CODE 

• CHANGED CODE 


UNMODIFIED 
RETESTED CODE 



njtvsvx 

MjUCMtntx Tl 



PRODUCTIVITY FACTORS 
INPUTS 

• CODE AND DATA BASE COMPLEXITY 

• CODING LANGUAGE AND DEVELOPMENT TOOLS 

• STAFF EXPERTISE AND EXPERIENCE 

• REQUIREMENTS COMPLEXITY AND STABILITY 

• USE OF MODERN PROGRAMMING TECHNIQUES 

• SPONSOR AND USER INVOLVEMENT 

• QUALITY AND MATURITY OF SYSTEM AND SUPPORT SOFTWARE 

• ADVERSE DESIGN CONSTRAINTS 

• ORGANIZATION AND WORK ENVIRONMENT 

• HARDWARE DEVELOPMENT CONCURRENCY 

• REAL-TIME VS NON-REAL-TIME PROCESSING 



OUTPUTS 


RESOURCE ESTIMATES AND DEVIATIONS 
CONFIDENCE FACTORS FOR SUCCESS 
SCALED STANDARD WBS SOURCE FILE 
PERT-PROCESSEDTASK LISTING 
GANH (SCHEDULE) CHART 
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BASIC MODEL CALCULATIONS 


EFFORT, STAFF MONTHS 
PRODUCTIVITY, KSLEC/STAFF-MONTH 
PRODUCTIVITY FACTOR 
TIME DURATION. MONTHS 
AVERAGE STAFF. PERSONS 
CPU TIME, HOURS 
DOCUMENTATION. PAGES 


W = L^/P^ 

P = PjL^'^ 

Pi = 

T = TiW*^ 
s = w/T = w^"‘^n‘i 

C = CiL^ 

D = Dll'* 


WHERE L I S COMPUTED FROM SIZE MODEL 

Aj ARE COMPUTED FROM PRODUCTIVITY FACTORS 

P . I, C,. D,ARE PRESET FROM MEASURED STATISTICS 
0 1 i I 
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CONFIDENCE CALCULATIONS 


RAYLEIGH-NORDEN-PUTNAMFORM: w = At'‘(K-w) 
SOFTWARE EQUATION: L = c 

P 

C0NF(L. W. T) = Pr {x<l, w< W, t <t} 

ASSUMES; 

(1) p, a r CHOSEN SO SOFTWARE EQUATION FITS POWER 

LAW FORMULAS 

(2) q/p SET TO GIVE OBSERVED TIME-EFFORTTRADEOFF 

IN DSN 

(3) pr { t ( w. X } , pr { w I X } , and pr are 

LOG-NORMAL DENSITIES 


1 


USAGE 




• PLANNING - CURRENTLY BEING USED TO ESTIMATE RESOURCE NEEDS 

OF ALL SOFTWARETASKS IN DEEP SPACE NETWORK CONSOLIDATION 

PROJECT 

• PROJECT CONTROL SYSTEM - INTEGRATION WITH WBS RESOURCE 

ACCOUNTI NG AND REPORTI NG SYSTEM CAN REDUCE CONTROL 
SYSTEM OVERHEAD 

• TECHNOLOGY EVALUATION - MODEL CAN POINT OUT WHERE RESOURCES 

ARE ACTUALLY GOING. AND WHERE ADVANCED SOFTWARE TECHNOLOGY 
I S NEEDED. AND WHAT BENEFITS FOR THAT TECHNOLOGY WILL BE 


CURRENT STATUS 


• PROTOTYPE IMPLEMENTED ON 8080-CP/M MICRO 

COMPUTER FOR EVALUATION 

• TECHNICAL REPORT NEAR COMPLETION 

• SUMMARY ARTICLE AVAILABLE 

• APPLICATIONS MANUAL BEING WRIHEN 


1 
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FUTURE PLANS 

1 


• MODEL CALIBRATION USING PERFORMANCE DATA FROM PROJECTS 

NOW IN PROGRESS 

• TRANSLATION AND INTEGRATION INTO PROJECT CONTROL SYSTEM 

• MODIFICATION TO ACCOMMODATE PROPOSED JPL SOFTWARE STANDARD 

STANDARD LIFE CYCLE AND PRACTICES 

• MODEL EXTENSION AND REFINEMENT TO INCORPORATE HUMAN 

ACTIVITY MODELS 

• UTILIZATION TO FORECAST TECHNOLOGY IMPACTS ON SOFTWARE 

PRODUCTIVITY 


1 
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TYPICAL MODEL USAGE 
EXAMPLE 


1 



TITLE: VERSION CONTROL EDITOR CDE: Angus Day 

ECR/ECO: e80.176 PROG. ID.: HUP-D2-OP-D.2 

SUBSYS: X21.6 Date Estimated: 14NOV80 

Model Data Version 1.3 31OCT80 


Answer the following items to the best of your estimation. 

1. How much new code is to be produced (completely new modules)? 

Maximum valuer kilo-lines executable source (99% confidence level)? 3.5 

Expected value, kilo-lines executable source? 3.3 

Minimum value, kilo-lines executable source(99% confidence level)? 3.1 

2. How much code exists in modules requiring modification? 

Maximum value, kilo-lines executable source(99% confidence level)? 6.9 

Expected value, kilo-lines executable source? €.6 

Minimum value, kilo-lines executable source (991 confidence level)? 6.3 

3. How much code will be deleted from these existing modules? 

Maximum value, kilo-lines executable source (99% confidence level)? .4 

Expected value, kilo-lines executable source? .3 

Minimum value, kilo-lines executable source (99% confidence level)? .2 

4. How much code will be added to these existing modules? 

Maximum value, kilo-lines executable source (99% confidence level)? .7 

Expected value, kilo-lines executable source? .6 

Minimum value, kilo-lines executable source (99% confidence level)? .4 

5. How much code will be changed in other ways in these modules? 

Maximum value, kilo-lines executable source (99% confidence level)? 1.2 

Expected value, kilo-lines executable source? .9 

Minimum value, kilo-lines executable source{99% confidence level)? .7 

6. How much code will be deleted as entire modules from existing code? 

Maximum value, kilo-lines executable source(99% confidence level)? 1.4 

Expected value, kilo-lines executable source? 1.3 

Minimum value, kilo-lines executable source(99% confidence level)? 1.1 

7. How much of the remaining existing code must be retested? 

Maximum value, kilo-lines executable source (99% confidence level)? 2.1 

Expected value, kilo-lines executable source? 1.9 

Minimum value, kilo-lines executable source(99% confidence level)? 1.5 

8. Expected percentage of code to be developed actually delivered 

(0-90, 91-99, ICO)? 91-99 

9. How many different kinds of input/output data items per 1000 lines of 

new or modified code(>80, 16-80, 0-15)? , 16-80 

10. Overall complexity of program and data base architecture 

(high, medium, low)? MEDIUM 

11. Complexity of code logical design (high, medium, low)? LOW 

12. What percent of the programming task is in Assembly language? 9 

13. What percent of the new or modified code must be storage-optimized? 9 
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14. What percent of the new or modified code must be timing-optimized? 

15. What percent of the total programming task is •easy’? 20 

16. What percent of the total programming task is ’hard'? 30 

17. When is work to start, on the(FRD/FDD, SRD, SDD) ? FRD/FDD 

18. What percent of the total program requirments will be established 

and stable before design, and will not be altered before delivery? 80 

19. What percent of the requirements are likely to change slightly before 

delivery, but will do so under baseline change control? 10 

20. What percent of the requirements are likely to change more drastically 

before delivery, but will do so under baseline control? 5 

21. Complexity of program functional requirements (high, medium, low)? LOW 

22. Expected user involvement in requirements definition 

(much, some, none)? MUCH 

23. Customer experience in application area(much, none, some)? SOME 

24. Customer/implementor organizational interface complexity 

(high, normal, low)? NORMAL 

25. Interfaces with other SW development projects or organizations 
(many, few, none)? 

26. Efficiency of implementing organization(poor , ok, good)? GOOD 

27. Overall implementation personnel qualifications and motivation 

(low, average, high)? HIGH 

28. Percentage of programmers doing functional design who will also 

be doing development {<25 , 25-50, >50)? 25-50 

29. Previous programmer experience wfth application of similar or greater 

size and complexity (minimal, average, extensive)? AVERAGE 

30. What is the average staff experience, in years, obtained from work 

similar to that required in the task being estimated? 6 

31. Previous experience with operational computer to be used 

(minimal, average, extensive)? MINIMAL 

32. Previous experience with programming language(s) to be used 

(minimal, average, extensive)? . ' ' MINIMAL 

33. Use of top-down methodology (low, medium, high)? HIGH 

34. Use of structured programmer team concepts ( low, medium, high)? HIGH 

35. Use of Structured Programming (low, medium, high)? HIGH 
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QA 

NO 

NONE 

70-90 

DEMAND 


36* Use of design and code inspections(low, QA# peer)? 

37. Classified security environment for computer (yes# # no)? 

38. Hardware under concurrent development (much# some# none)? 

39. Percent of work done at primary development site 
(<70# 70-90# >90)? 

40. Development computer access mode (remote# scheduled# demand)? 

41. Percent of development computer access availability (<30# 30-60# >60)? 30-60 

OK 

' OK 

KINIKAL 
NO 


42. Quality of SW development tools and environment (poor , ok, good)? 

43. Maturity of system and support software (buggy, ok, good)? 

44. Overall adverse constraints on program design 
(severe, average, minimal)? 

45. Is the program real-time, multi-task(chief ly , some, no)? 

46. SW to be adaptable to multiple computer configurations or environments 
(yes# # no)? 


47. Adaptation required to ch^ge from development to operational 
environment (much # some# minimal)? 


MINIMAL 
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Estitaated Overall ParameterB: 


^average value 

n-sigma -l-slgaa 


.Adjusted Lines of code- 6182 SLEC 
6280 

Effort- 26.5 person-months 

Stiff productivit|-S33 SLEC/staff-month 

404 

Duration- 15.7 “onths 
19.0 12.9 

Avg. Staff- 1.7 ^ ^ 

Documentation- ^^*$19. 4K 

CoIJIter CPU time- 319 hours JO.OK 
478 


$13. 4K 

$0.0K 


im#. these figures to arrive at Effort, 
inclua. fctoc. to p.o.1 
and confidence levels. 


Duration^ and Staffing 
de acceptible risk' 


Values specified are: 6 

Kilo-lines of code- 3 

Effort (person-months) : 1 

Duration (months) : 

Average staff (persons) : 

,.t th. ooob,.. VOO b.». .ot...0, . 

Th. 1...1 Of confldonc. in d.Uv.rioo «162 »"•» »' 
on-time and within resources- 33 %. 


Is output to be saved in a file? 
Name of output file to be created; 
Schedule start date; 


£s;-i:s.K K -» “ 


1- Gantt Chart 

2- PERT data, 132 width 

3- PERT data, 80 width 


A-file 

B»line printer 


1B,2B,3A 


Chclcc \C) 
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I TITLES VERSION CONTROL EDITOR 
\ ECR/ECOi eBO.176 
I SUDSYS: X21«6 


EFF 


I CODE 


TASK 


: WHO 


ORT s t EARLY 

1 t DAY DATE 


* 0 . 

• 1 . 

* I.l 

* 1.2 

I 1.2.1 

• 1 . 2.2 

• 1.3 

• 1.4 

• 1.5 

I 1.5.1 

• 1.5.2 

• 1.6 
• 2 . 

• 2.1 
• 2.2 
I 2.2.1 
• 2 . 2.2 

• 2.3 
•3. 

• 3.1 

• 3.2 

I 3.2.1 

• 3.2.2 
I 3.3 

• 3.4 

M. 

I 4.1 
I 4.1.1 

• 4.1.2 
I 4.1.3 
I 4.1.4 
I 4.1.5 
I 4.1.6 
I 4.2 

• 4.2.1 

1 4.2.2 

\ 4.2.3 

• 4.3 

• 4.4 

• 4.4.1 

• 4.4 2 

• 4.4.3 

I 


START 

Sya Plans# Reqts# & Design 
Define Subsys Reqts 
FRD 

Write all sections 
Edit and release FRD 
Level B Review 
Define Sys Architecture 
FDD 

Write all sections 
i:dit and release 
Level C Review 
SW Planning and Reqts 
Define Software Reqts 
SRD 

Write all sections 
Edit and release 
Level D Review 
SW Architecture and Design 
Define SW architecture 
SDD 

Write all sections 
Edit and release 
System Interface Design 
Level C Review 
SW Detailed Design li Prod 
SSD 

write Sections 1#2»3 
Write Section 4 
Write Section 5 
Write Section 6 
Write Section 7 
Edit and release 
SOM 

write preliminary draft 
Complete all sections 
Edit and release 
High-level Design Review 
Module Production 6 Integ 
Executive and control 
I/O Modules 
Interface handlers 


0 

t : 

0 

0 

t : 

32 

15 

: t 

0 

0 

X X 

13 

4 

: : 

0 

2 

: 1 

12 

3 

: : 

13 

18 

: 1 

15 

0 

: 1 

30 

4 

: X 

15 

2 

1 X 

29 

3 

: X 

30 

0 

X X 

58 

25 

1 X 

32 

0 

X X 

56 

4 

X X 

32 

2 

X 1 

55 

2 

X t 

56 

0 

: X 

93 

31 

X X 

58 

0 

X X 

91 

4 

X 1 

58 

2 

X X 

90 

22 

X X 

58 

2 

: X 

91 

0 

X X 

273 

0 

: X 

315 

6 

X X 

93 

18 

X X 

93 

43 

X X 

107 

4 

1 X 

93 

9 

X 1 

93 

6 

X X 

312 

0 

X X 

314 

8 

X X 

107 

9 

X X 

133 

4 

I X 

312 

2 

X X 

131 

0 

X X 

271 

18 

X X 

113 

18 

X I 

133 

18 

X X 

151 


17NOV80 

9JAN81 

17NOV80 

BDECeO 

17NOV80 

5DEC80 

8DEC80 

1QDEC80 

7JAN81 

10DEC80 

6JAN81 

7JAN81 

16FEB81 

9JAN81 

12FEB81 

9JAN81 

llFEBBl 

12FEB81 

6APR81 

16FEB81 

2APRB1 

16PEB81 

1APR81 

16FEB81 

2APR81 

22DEC81 

23FEBB2 

6APRB1 

6APR81 

24APR81 

6APR81 

6APR81 

18FEBB2 

22FEB82 

24APR81 

2JUN81 

18FEB82 

29MAY81 

18DEC81 

4HAY81 

2JUN81 

26JUN8I 


CDEi Angus Day 
PROG. ID.: HUP-D2-OP-D.2 

STATUS AS OPi 14NOV80 


LATE 

DAY DATE 

XX 
X X 
X X 

FINISH 

EARLY X LATE 

DAY DATE x DAY DATE 

XX FLOAT 
XX TIME 
X X DAYS 

0 

17NOV80 

X X 

0 

17NOV80 

0 

17NOV80 

X X 

0 

32 

9JAN81 

X X 

32 

9JANB1 

32 

9JAN81 

X X 

0 

0 

17NOV80 

X X 

10 

3DECB0 

10 

3DEC80 

X X 

0 

13 

8DEC8Q 

xt 

13 

8DEC80 

13 

8DEC80 

X X 

0 

9 

2DEC80 

X X 

3 

20NOV80 

12 

5DEC80 

X X 

9 

12 

5DEC80 

X X 

13 

8DEC80 

13 

8DEC60 

I X 

0 

13 

8DEC80 

X I 

IS 

10DEC80 

15 

10DEC80 

X X 

0 

15 

10DEC80 

X 1 

27 

31DEC80 

27 

31DEC80 

X X 

0 

30 

7JAN81 

X X 

30 

70ANB1 

30 

7JAN8I 

XX 

0 

27 

31DEC80 

X X 

17 

12DEC80 

29 

6JAN81 

X X 

12 

29 

6JAN81 

X X 

30 

7JAN81 

30 

7JAN81 

X X 

0 

30 

7JAN81 

X X 

32 

9JAN61 

32 

9JAN81 

X X 

0 

58 

16FEB81 

X X 

58 

16FEB81 

58 

16FEB81 

X X 

0 

32 

9JAN81 

X X 

50 

4FEBB1 

50 

4FEB81 

X X 

0 

56 

12FEB81 

X I 

56 

12FEB81 

56 

12FEB81 

X X 

0 

53 

9FEB81 

X X 

34 

13JAN81 

55 

llFEBBl 

XX 

21 

55 

llFEBBl 

X X 

56 

12FEB81 

56 

12FEB81 

X X 

0 

56 

12FEB81 

X X 

58 

16FEB81 

58 

16FEB81 

XX 

0 

93 

6APRB1 

X X 

93 

6APR81 

93 

6APRB1 

XX 

0 

58 

16FEB81 

X X 

81 

19MAR61 

81 

19HAR81 

X X 

0 

91 

2APR81 

X X 

91 

2APR81 

91 

2APR81 

XX 

0 

88 

30MAR81 

X 1 

60 

18FEB81 

90 

lAPRSl 

X X 

30 

90 

lAPRBl 

X X 

91 

2APRB1 

91 

2APR81 

XX 

0 

78 

16NAR81 

X X 

70 

4HAR61 

90 

1APR81 

X X 

20 

91 

2APR61 

X X 

93 

6APR81 

93 

6APR81 

X X 

0 

273 

22DEC81 

I X 

273 

22DEC81 

273 

22DEC81 

X I 

0 

327 

11MAR82 

X X 

315 

23FEDB2 

327 

11MAR82 

X 1 

12 

268 

15DEC81 

XX 

96 

9APR81 

271 

18UEC61 

X X 

175 

93 

6APR81 

I X 

107 
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SOFTWARE COST/RESOURCE MODELING: 
Software Quality Tradeoff Measurement 

R. W. Lawler 

Boeing Aerospace Company 


1.0 INTRODUCTION 

In the early 7C’s the concept of software quality focused on the property of software reliability - 
freedom from errors. Subsequent research [McCall (1)) has identified eleven other software qual- 
ity factors and has developed a system of metrics to predict/assess the degree of presence of the 
various qualities in software. The metrics have been validated for some quality factors. However, 
the concepts and validation have focused primarily on the software subsystem and have largely 
ignored some of the total system aspects such as the computer hardware, operating system, com- 
munications network, and other noncomputing elements of a total system. This paper develops 
a conceptual framework for treating software quality from a total system perspective. Examples 
are given to show how system quality objectives may be allocated to hardware and software; to 
illustrate trades among quality factors, both hardware and software, to achieve system perform- 
ance objectives; and. to illustrate the impact of certain design choices on software functionality. 


2.0 STATUS OF SOFTWARE QUALITY FRAMEWORK 

Figure 2.0-1 summarizes the definitions of software quality developed by McCall. Criteria and 
metrics have been defined for many of these quality factors. McCall validated metrics for four 
quality factors, namely 

reliability (design and implementation); 
maintainability (design and implementation); 
flexibility (implementation; and, 
portability (implementation). 

The sample size used for validation was too small to establish acceptable confidence limits. The 
significance of his work was that relationships between metrics do exist and correspond to intui- 
tion. There is ample need to perform similar validations for these and other quality factors. 


3.0 THE SYSTEM QUALITY PERSPECTIVE FOR DISTRIBUTED SYSTEMS 

The Software Quality Perspktive can also be formulated from a total system point of view. That 
is, computer hardware, communications^etwork, operating system, and even non computing ele- 
ments are considered in a system perspective on Quality. 

In fact a major problem that plagues the automated system design process is the lack of a frame- 
work for including software in the process of allocating system requirements and quality goals. 

A meaningful and practical software quality framework for distributed systems must address this 
problem. Three examples of this impact are: 

(1) System quality factors may change the emphasis of a software quality factor, criterion, 
or metric by either increasing or decreasing its importance in the context of distributed 
systems. For example, software fault tolerance receives increased emphasis in a distrib- 
uted system, whereas in a single-processor system the emphasis is usually placed on 
error recovery. 
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function with reiiuireil precision. 

The amount of computing resources and code required by a pro- 
grain to pertonn a t unction. 

l*!\tcnl to which access to Si>ltwarc or ilata hy unaiithori/cvl pcr5U.^ns 
can be controUeil. 

i:iYort required to learn, operate, prepare input, and interpret out- 
put of a prognim. 

Hffort required to locate and fi\ an error in an operational program. 

I’lTort required to test a pmgram to incure it pertonus its intended 
function. 

r.ffort required to modify an operational program. 

i:ffort required to transfer a program from one hardware contigura- 
lion and/or software system environment to another. 

l-xtent to which a program can be used in other applications - 
related to the packaging and scope of the functions that programs 
perform. 

I’ f fort required to couple one system with another. 

1‘igure 2,0-1. Software Quality bactor nefinitions 
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{1) System tjuality factors may require moilification or additions to si>ft\vare quality factor 
concepts. For example, the system quality of survivability requires the addition of a 
software quality of survivability. 

(3) Requirements for system quality may lead to increased s^dtware functionality. F’or ex- 
ample, the operating; system may be required to incor|x>rate dynamic reconfigurabilit> 
to achieve system reliability requirements. 

There is a need to define sy stein Quality 1‘actors and Criteria. F’igure 3.0-1 shows a partial com- 
pilation of these Quality Factors and Criteria. I'ij:ure 3.0-2 pro|\)ses definitions for some of the 
system quality factors and criteria. 

The figures show that 

(1) there are additional quality factors for systems e.g. scifety and availability for sy.stems 

(2) the system quality perspective adds a large number of new criteria, e.g. distributedness 

(3) the definitions of system ipiality factors and criteria may iliffer frtnn the software defi- 
nitions. e.g. maintainability 

Tliere is a relationship between system quality factors and criteria and .software quality factors 
and criteria. Figure 3.0-3 gives examples of this correspi^ndence. 

F'or example, the system quality factor of availability implies software quality factors of correct- 
ness and maintainability. High quality in these factors insures that the system will seldom fail 
due to a software error and if one occurs the fault will be corrected ipiickly. Availability in a 
distributed system als^> implies software reliability criteria of fault recovery and fault tolerance. 
These properties insure that the system continues to run if a processor or communication link 
fails, and that once repair is made the failed com|xment rejoins the network quickly. 

Some system quality factors correstxmd exactly to their software quality counterpart, e.g. integ- 
rity. But other quality factors do not. F’or example, system jxirtability implies low power, light 
weight, and compact. These are not software qualities, but they imply that there are constraints 
on the software in the form of low |H>wer and low speed logic; limited facilities for data storage, 
need for firmware rather than s^)ftware; limited facilities for data entry and display; and electro- 
magnetic communication links. In this envinmment the quality factors of efficiency, usability, 
and integrity would be emph;isi/ed. 

If a system is highly distributed with respect to the control logic of the sv>ftware. testability of th 
Si^ftware system and res|x>nse time of the proces.sing will be inqxutant considerations. 

One of the major impacts of systen. quality factors is tlial they place reiiuirements for increased 
functionality on the st>ftware. Figure 3.0-4 illustrates the impact of a set of system Quality 
factors on a distributed operating system. Collectively the eight factors require the addition of 
Si>ftware in eight functional areas. 

The considerations that are made in allocating a system i|uality factor to hardware, software, 
firmware, and operating system for a distributed system may differ frxun those for a single pro- 
cesst>r system. F'igures 3.0-5 and 3.0-O are examples of allocating system quality factius o\ . 
efficiency and reliability to system comixments. 
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Frror Tolerance 

Flexibility 

Parallelism 

Simplicity 

Testability 

Taskinji L'tYieieney 

Modularity 

Portability 

Communications FiYieicne\’ 

Cienerality 

lls;ibility 

Multilevel Security 

Fxpandability 

F.tTieieney 

Homogeneity 

Instnimentation 

lniei:rity 

Virtuality 

Self Deseriptiveness 

Interoperability 

Fiuiuranee 

Fxecution Ffficiency 

Survivability 

Reconfigurability 

Access Control 

Reusiibility 

Dispersion 

Access Audit 

Safety 

Indepeiulence 

Operability 


Ctraeeful Degradation 

Training 


C'ommunieativeness 

Conciseness 


Commonality 
Damage Potential 

Warnings 


I'iguro 3.0-1. ('aiuliiiatc System Quality I'actors aiul Criteria 


System 

F'actor or Criteria 

Definition 

-Availability 

1 - M n R/MTUl- 

where M ITR is Mean Time to Repair 

M FBI* is Mean Time Between Failure 

Survivability 

Probably that a distributed system still performs essential functions 
after one or more nodes and communication links are desta>yed or fail. 

Distributedness 

Degree to which communication connected process^irs, data bases, 
users, and/i>r priHcsses are geographically or logically separated. 

Maintainability 

Mean fime to repair 


I'iiiure 3.1V-2. Caiulklate Deiiiiitious of System Quality I'aetorsaiul t'riteru 
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System 

Software 

Quality Factors/Criteria 

Quality Factors 

Criteria 

Availability 

Maintainability 

Fault Tolerance 


Correctness 

Fault Recovery 

Integrity 

Integrity 

No Change 

Portability 

Kffieieney 

Usability 

Integrity 

No Change 

Distributiveness 

Testability 

Response 

Time 


Fijiuro 3.0-3. Correspoiulcnce of System and Software Quality. Factors am! Criteria. 



I'ieure 3.0-4. System QualilN !■ actors Impact on Opera! inj: System Functionality 
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HARDWARE 

FIRMWARE 

OP. SYSTLM 

APP.S/W 

SINGLE 

PROCESSOR 

High Speed 
Components 

Increase Speed 

Priority Tasking 

Optimized 

Code 

DISTRIBITED 

SYSTEM 

Low Speed 
Components 
High-Speed 
Communication 

Increase Speed 

Synchronization 

Fxploitation of 
Parallelism 


Figure 3.0-5. .Allocation of Efficiency to System Components 




RELIABILITY 




HARDWARE 

FIRMWARE 

OP. SYSTEM 

APP. S/W 

SINGLE 

PROCESSOR 

High Reliability 
Parts 

No Impact 

Fault Recovery 

No Impact 

DISTRIBITED 

SYSTEM 

Low Reliability 
Parts Redundancy 

No Impact 

Fault Tolerance 
Fault Recovery 

No Impact 


Fiuurc 3.0-ti. Alkication of Reliability to System Components 


This type of analysis 
systems with respect 


liichlights the difference between single proeessor systems and distributed 
to systent and software quality. It also identities the trades among the sys- 


tem compmtents. 


4.0 MAN.XGINC. SOI-TWARF QUALITY .-\CQU1S1T10N 

The system Quality perspective intluences the activities of a software acquisition manager. There 
are several issues namely 

( 1 1 the issues that arise between a system ai-quisition manager and a software acquisition 
manager in allocation of quality goals to software; 

( "I the issues that arise between a software aciiuisition manager and a hardware acquisition 
manager in ensuring that hardware and software implementations ol the quality goals 
jirc compaiiblc; 

the issues that arise during a competitive phased acquisition between the s<iftware ac- 
quisitiou manaircr aiul the contractors: 


R. U%»tex 
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(4) the issues that arise when an existing distributed system is being modified or expanded 
to include new tunctions; 

(5) the issues that arise when a distributed system is being modernized. 

These issues are illustrated by the foUowing seenarios. 


4.1 System Availability Scenario 
nm,e nun.be, of 

and compatible design strategy. 

T,,c»f.wa,emuna,o,b,.ub,do..n.be^ 

even though there are hardware faults. 

. ■ .Ir.c .tv rel ited to hardware maintainability because the system is not available 

Fault recovery is dose y related to comnuter system restarted, and the data processing 

until the hardwai- tault is J j ha^j^are failure occurred. Recovery may in- 

system notification to system users etc^ The 

dude restoration ot tiks, r r^ult recovery must be compatible with the hard- 

relative en.pl, avis fte ^s^em is suet, thai a sinple hardware 

7ie"e1us:n"y,.e“ STthen the sSf.ware should en,phasize fault recover,- rather than 
fault tolerance. 

The software JLn« after UstnSdevX"^^^^^ 

rectness. It is ^oM be an important factor for system maintainability, 

1 V helimTbdwwn identificLion of a new requirement or function and the incorpor- 
St- that ;r::t^;rin;o me system is included in the system availability concept. 

’ » • hiiJiTft tmdeoffs between correctness and maintainability 

I,eS”%S\‘Seo,'?r,S eoisideralioh of the feasible software expenditure pronie 

and the desired initial operation dale. 

I • th*^ <nftware acQuisition manager establishes a feasible range 
In a eontpetlllve phased ae.iuis.tion strjtwy for meeting the budget range. Suppose 

and tbe eumpetlng eonlraelors "Se that meets lor, of the tenuitemems. To 

i\'l'abm!’y ™dTTbtm; ratl’rT:::- 2~Tan' ' 

there is a good ehanee that the pn-pused soliwat. cjn t be used j . 
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Figure 4 0-1 summarizes the issues exposed by this example and the relationship ^ 

wftware acquisition manager and the system acquisition manager, the hardware acquisition man- 
ager, and the competing contractor. 


4.2 Multifunction System Scenario 

One wav of characterizing distributed systems is the term “multifunction system". That is, the 
svstem Lrforms multiple functions; the responsibiUty for performing those functions is usually 
d^ribut^d among several physically separated processors or processing complexes; and the execu- 
rion of any one function will normally require cooperation among several physicaUy separated 
processors/complexes with distinct capabilities. This is analogous to any large system with mult - 
pie subsystems cooperating to perform system functions. 

From this Doint of view, quality requirements for the various subsystems are derived by; exam- 

fZItion to each subsystem invoLed in the etteculion or that fnnet.on;and then combmmg the 
allocations for one subsystem to determine its quality requirement(s). 

For example a navigation function for a military application being irnplernemed in a distributed 
svstem may have high reliability, efficiency, and correctness requirements. These r^uu-ements 
wm be aTocated differently to different parts of the system. The correctness requnement would 
most likely be allocated to the software and possibly to the processor (e.g. the need for fl g 
point precision) and the sensors (e.g. precision and accuracy). 

The reliabilitv and efficiency requirements will be allocated among the system elements in line 
with performing navigation; a certain amount of accuracy would be r^J^red of the sensors, the 
perion HI b b sensors to the processors would most likely be redundant for re- 

SSy S riS hdc fc, cmclcncy: .he pmces»m would also IMy 

bf “undant and also requim a minimum pmeessing speed; and .he soflwam wdl .equue some 
iype of resource management scheme to insure navigation function reliability. 

If nftPr the svstem is implemented a weapon delivery function is added which uses some of the 
mme selrsls saLT^ "" syslem qua.- 

itv will be impacted. For example the addition of this function may overload the commun^a- 
Sn svstem Additionally the weapon delivery function will have quality requirements appropri- 
ate to^ts needs and different than the quality requirements for the navigation function. More- 
over after the new quality requirements are allocated the various elements, it will be necessary- 
to combine the requirements from the different functions in order to arrive at the quality re- 
quirements for individual elements. 

It IS then necessary to perform trade studies in order to iteratively arriv^ at an optimum dcsi^ 
clint on For example the reliability allocation to the software might be decreased (with a cor- 
SjSing decrease in cost) if management of the redundant communication link and any 
•inf sensors was placed in microprocessors which front-ended the mam processors provided th 
thi- cost to implement and maintain microprocessors with the rc-quired reliability did not exceed 
the cost budget for that subfunction. 


5.0 SOFTWARE QUALITY IMPACT ON DEVELOPMENT COST 


From Figure 2.0-1 it is dear 
life exists will be reduced. It 


that the more qualitv is built into the software, the more operational 
is also clear that there are trade offs among software quality factors - 
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Acquisition 

Manager 



FIGURE 4.0-1 Trades and Allocations Relevant 
to System Availability Quality 
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for example it is generally accepted that increasing software efficiency will reduce its portability, 
flexibility, and usability. But the impact of software quality on development cost has not been 
explored. Currently software costing models do not account for the impact of software quality 
on development costs. 

Figures 5.0-la through 5.0-ld illustrate four alternate theories for this impact. They are; 

(1) Software quality if productive — The greater the degree of software quality the lower 
development cost. i.e. Software quality is the key to increasing software development 
productivity; 

(2) Software quality is free - Software development costs and software quality are inde- 
pendent. The level of quality attained is a consequence of the development method- 
ology. In this case the interest in measuring software quality is partially directed at 
determining a superior development methodology: 

(3) Software quality costs - Enhance quality is achieved only at increased cost. In this 
case there is a tradeoff between degree of quality and life cycle cost. Quality measure- 
ment is directed at determining when to stop increasing quality; 

(4) Software quality is a combination of (1), (2), and (3). i.e. In the low range theory (1) 
holds, in the mid range theory (2) holds, and in the high range theory (3) holds. 

Intuitively Theoiy (4) appears to be more realistic than the other three because 

(1) If a software system ranks extremely low in maintainability and reliability during the^^ 
development process it will take an “infinite” amount of time to correct an “infinite” 
number of errors. Even a small increase in quality will have a beneficial effect; hence 
the direction of section one is correct. 

(■*) A larce number of programs of equivalent size lie within a narrow cost range. The 
quality of the programs vary widely, i.e. Section 2 of the curve has the correct slope. 

(3) When programs utilize a large percentage of CTU time and/or memory capacity the cost 
of the software grows exponentially. But when utilization is below 50% efficiency is 
not a concern and traditionally software costs are reduced dramatically. Figure 5.0-2 
illustrates a PRICE -S estimation of software costs as utilization increases from 60% to 
95%. The curve was derived from data in Gaffney 80. 

The rising portion of curve 5.0-ld is of interest because that is where trades between develop- 
ment cost and operational life costs are made. If all software quality factor curves are exponential 
like the PRICE -S model. 


6.0 CONCLUSIONS &. RECOM.MENDATIOSS 

Measurement of the Qualitv of Software for Specific factors that are important in the operational 
phase provides a rational method of selecting among alternative off-the-shelf software packages. 
This technique will satisfy user concerns. But the developer needs to be able to determine the 
amount of qualitv that can be built into the software cost effectively. The systems approach can 
indicate the relative emphasis to be placed on software quality factors. But models are needed 
to help determine when it is cost effective to terminate improvement of software quality factors. 
An operational approach, such as tracking the marginal improvement in software quality can be 
used as a stopgap measure. 

lOoflJ 






CORE/CPU UTILIZATION 


Figure 5.0-2 Impact of Utilization on 

Software Development Cost PRICE-S 


R.U»1ci 
Boeing Aerospace 
12ofl3 


The cost development curve in the high quality range appears to be exponential for a single 
quality factor. But models for determining the development cost impact of a mix of quality 
factors requires a model that factors in the correlations among quality factors. 
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SOFTWARE RELIABILITY: 

Application of a Reliability Model To Requirements Error Analysis 

J. Logan 
TRW 


Although requirements errors were identified several years ago as a major source of software 
problems encountered in software integration and testing, most reliability studies have been con- 
cerned with other issues and have made little contribution to solving requirements problems. 
Application of a software reliability model having a well defined correspondence to computer 
program properties to requirements error analysis has identified requirements error categories re- 
latable to program structural elements and to their effect on program execution. 

Analysis of B-5 type software requirement specifications has confirmed that errors of these types 
do occur. 

The software reliability model represents in precise terms the software reliability definition: the 
probability that a program will compute required output values under specified conditions. In 
the model, the specified operational conditions are represented in terms of the set E of all possi- 
ble inputs, the set Ej of input values for a specific execution, and the probability of choosing Ej 
as an input. Correct comput^ion is defined as the actual computed value of (Ej) being equal 
to the required output value uEj) within a specified tolerence 6j. 

The model represents a software requirements specific^ion in terms of the function T the pro- 
gram is required to computron the input domain set E. The requirement specification generally 
will partition the input set E into input subsets (input domain partitions) and the function 
into functions fj on Uj. These requirements elements correspond to program elements: 

• E corresponds to the program input domain E 

/N ys 

• Gj correspond to input domain subsets Gj 

• tj corresponds to the logic path Lj 
executed for inputs from Gj and 
which computes the function fj on Gj 

Requirements errors arise from: 

• incomplete specification of 

• the input domain E, 

• the input domain partition subsets G^, or 

• the functions fj on Gj. 

• incorrect specification of these same items or 

• association ot a tunction tj with the wrong input domain partition Gj.. 

The B-5 software requirement specification format is compatible with the model, for it presents 
the requirement specifications for a program module in terms of: 

J. Itygui 
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• Inputs 


• Specifying H and Gj 

• Processing 

• Specifying the functions 1j to be tximputeil 

• Outputs 

• Specifying the output values Ijtlij) 

Tlie requirement error categories provide a systematic means for examining the requirement 
specification for errors, aiding quick identification of them. 

Here is a simplified example of a B-5 type requirement specification for a program module named 
F.vent Processor. Although this example is not a real example, it closely approximates the prob- 
lems found in actual requirements review.s. 

The livent Processor is required to log event occurrence and select for each event occurrence the 
task to be performed in acairdance with prescribed action criteria. 

The module is required to process two inputs, the event identifier and a task action table. Two 
processing functions are specilied: 

• on occurrence of an event, select the task meeting task action criteria, and 

• |K)st the event, time of event occurrence, and the task selected in the event log. 

One output, the iqxlated event log. is specilied. 

Some of the problems in this requirement specification are shown on the following viewgraphs. 

The first problem identified is incomplete specification of the input Task Action Table. Only 
the name of this input is specified. While the name indicates it is a table, the structure of the 
table and the range of values allowed as elements of the table are not specified. Presumably, 
definition of this tahle is left to the software designer. 

The second problem identified is incomplete specification of a processing function. The task 
action criteria, presumably algo.ithms for selecting the task, are not defined. Possibly the incom- 
pletely specified task action table contains a tabular representation of the criteria, again for the 
software designer to invent. While this may result in a correct design, generally, the software 
designer understands the problem the program is intended to compute less well than the require- 
ment specifier and therefore is more likely to design a program computing the wrong problem. 

Tlie third problem identified is reference in a processing function to an input variable time not 
on the input list. . 

The fourth problem identified is a missing function. Presumably Ugain incomplete input specifi- 
cation) the task aetion table contains an entry for each expected event. The missing tunction is 
the processing to be performed in an event identifier not present in the task action table is pre- 
sented to the program. 

J. Login 
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TIk'sc four problems arc not the only problems in this specifieation. but time does not permit 
discussion of the others. 
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TIU^ VlHWGRAPn MATERIALS 
for the 

J. LOGAN PRESENTATION FOLLOW 
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RELIABILITY MODEL APPLICATION TO REQUIREMENTS ERRORS 

REQUIREMENTS ERRORS 

• MAJOR SOURCE OF SOFTWARE PROBLEMS 

RELIABILITY MODEL 

• IDENTIFIES REQUIREMENTS ERROR CATEGORIES 

ANALYSIS OF B-5 SPECIFICATIONS 

• CONFIRM ERROR CATEGORIES 


J. Logta 
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SOFTWARE RELIABILITY MODEL 


SOFTWARE RELIABILITY DEFINITION 

• PROBABILITY THAT PROGRAM WILL COMPUTE REQUIRED OUTPUT 
VALUES UNDER SPECIFIED CONDITIONS 

SPECIFIED OPERATIONAL CONDITIONS 

• E : SET OF ALL POSSIBLE INPUTS 

• E,: INPUT TO SPECIFIC EXECUTION 

• P, : PROBABILITY OF CHOOSING E, 

CORRECT COMPUTATION 

• F(E,); ACTUAL COMPUTED VALUE 

• fie,); required VALUE 

• IFIE,) - FIE,) 1< e,: CORRECT COMPUTATION 
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MODEL REPRESENTATION OF SOFTWARE REQUIREMENTS 


PROGRAM IS REQUIRED TO 

• COMPUTE FUNCTION F FOR ALL INPUTS IN E 

REQUIREMENTS PARTITION 

• E INTO SUBSETS Gj 

• ^ INTO FUNCTIONS Fj ON Gj 

REQUIREMENTS ELEMENTS CORRESPOND PROGRAM ELEMENTS 

• E CORRESPONDS TO INPUT DOMAIN E 

• Gj CORRESPONDS TO INPUT DOMAIN SUBSET Gj 

• Fj CORRESPONDS TO LOGIC PATH Lj COMPUTING FUNCTION Fj ON Gj 

REQUIREMENT SPECIFICATION SPECIFIES 

• E 

• G] 

• Fj ON Gj 
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REQUIREMENTS ERRORS ARISE FROM 


INCOMPLETE SPECIFICATION OF 

• t 

• “Fj ON 

INCORRECT SPECIFICATION OF 


/\ 

• E 

• 

• Fj ON Gj 


ASSOCIATION OF 
• Fj WITH 6k 
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APPLICATION TO B-5 SOFTWARE REQUIREMENT SPECIFICATION 

B-5 FORMAT IS COMPATIBLE WITH MODEL 

• INPUTS CORRESPONDS TO E AND Gj 

• PROCESSING CORRESPONDS TO 

• OUTPUTS CORRESPONDS TO Fj(Ej) 

REQUIREMENT ERROR CATEGORIES 

• AID QUICK IDENTIFICATION OF ERRORS 
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B-5 REQUIREMENT SPECIFICATION EXAMPLE 


NAME: EVENT PROCESSOR 

• REQUIRED TO LOG EVENT OCCURRENCE AND SELECT TASK IN 
ACCORDANCE WITH ACTION CRITERIA 

INPUTS 

• EVENT ID 

• TASK ACTION TABLE 

PROCESSING 

• ON OCCURRENCE OF EVENT, SELECT TASK MEETING TASK ACTION 
CRITERIA 

• POST EVENT, TIME OF EVENT OCCURRENCE, AND TASK SELECTED IN 
EVENT LOG ' 

OUTPUTS . 

• UPDATED EVENT LOG 


J. Logan 
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INCOMPLETE SPECIFICATION OF INPUT 


NAME: EVENT PROCESSOR 

• REQUIRED TO LOG EVENT OCCURRENCE 

accordance with action criteria 


AND SELECT TASK IN 


INPUTS 

• EVENT ID 

• TASK ACTION TABLE CnOT DEFINED) 


PROCE^^NG^C^rrEnce of event, select task meeting TASK ACTION 
CRITERIA 

• POST EVENT. TIME OF EVENT OCCURRENCE. AND TASK SELECTED IN 
EVENT LOG 


OUTPUTS 

• UPDATED EVENT LOGS 
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INCOMPLETE SPECIFICATION OF PROCESSING FUNCTION 

NAME: EVENT PROCESSOR 

• REOUIRED TO LOG EVENT OCCURRENCE AND SELECT TASK IN 
ACCORDANCE WITH ACTION CRITERIA 

INPUTS 

• EVENT ID 

• TASK ACTION TABLE 

PROCESSING 

• ON OCCUR RENCE OF EVEN T. SELECT TASK MEETING TASK ACTION 
CRITERIA (NOT DEFINED) 

• POST EVENT. TIME OF EVENT OCCURRENCE, AND TASK SELECTED IN 
EVENT LOG 

OUTPUTS 

• UPDATED EVENT LOG 
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PROCESSING REFERENCE OF VARIABLE NOT ON INPUT LIST 


NAME: EVENT PROCESSOR 

• REQUIRED TO LOG EVENT OCCURRENCE AND SELECT TASK IN 
ACCORDANCE WITH ACTION CRITERIA 

INPUTS 

• EVENT ID 

• TASK ACTION TABLE 

PROCESSING 

• ON OCCURRENCE OF EVENT, SELECT TASK MEETING TASK ACTION 
CRITERIA 

• POST EVENT, TIME (NOT ON INPUT LIST) OF EVENT OCCURRENCE, 
AND TASK SELECTED IN EVENT LOG 

OUTPUTS 

• UPDATED EVENT LOG 
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PROCESSING FUNCTION NOT DEFINED FOR INPUT PARTITION 

NAME; EVENT PROCESSOR 

• REQUIRED TO LOG EVENT OCCURRENCE AND SELECT TASK IN 
ACCORDANCE WITH ACTION CRITERIA 

INPUTS j 

• EVENT ID 

• TASK ACTION TABLE 

PROCESSING 

• ON OCCURRENCE OF EVENT, SELECT TASK MEETING TASK ACTION 
CRITERIA 

• POST EVENT. TIME OF EVENT OCCURRENCE. AND TASK SELECTED IN 
EVENT LOG 

• (PROCESSING FOR EVENT ID NOT IN TASK ACTION TABLE IS NOT 
DEFINED) 

OUTPUTS 

• UPDATED EVENT LOG 


3. U>gan 
TRW 
14of 14 



Optiimun 


SOFTWARU RltU ABILITY; « 

Maintenance Policies for Hardware-Soltware Systems 


Ainrit L. Goel and J. Soenjoto 
Syracuse University 
Alan Sukert 

Rome Air Development Center 


The objective of tliis A markov model 

and se)ftware components | \ , nviintenanee times (of both hardware and st>lt ware 

Distributions of the Next we explain the maintenance and L»L>je rate 

components') are assumed to operational performance measures sueh as 

nivinu some laetors on whieh they ‘ ,• successful operation tor a specitied tune, sys- 

distribution of time to next ‘a'li'r^- P''^ t',iiures bv a specified time are developed next and 
torn availability, distributuui ot numl er o U - . ^ cost of tailures. system 

studied. Afterwards, we develop a ex, t problem as a non-hnear optmu- 

p;o^ier;:;;;ror^- ^ 

the optimum maintenance polieies. 
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ECONOMICALLY OITIMUM HARDWARE/SOFTU'ARE MAINTENANCE STRATEGIES 

by 

Amrit L. Goel*. Jopie B. Soenjoto^ Alan N. Sukert^ 


Continued increase in the use of computer systems in a wide variety of applications has necessi- 
tated a greater emphasis on the development and deployment ot cost-effective and reliable hard- 
ware/software systems. Several models have been developed and tested during the past seven 
years to quantity the perlormanco ol such sot t ware systems. 

In this presentation, we summarize our previous results and then present a Markovian model tor 
the operational phase of a hardware/software system. Several performance measures, such as 
average system availability, and number of software and hardware failures by time t, arc developed 
and studied. A cost model is proposed which incorporates the cost of lailures, system unavail- 
ability and maintenance. Optimum maintenance policies are computed by lormulating the pro 
lem as a nonlinear optimization problem with or without constraints on repair rates and system 

availability. 


*Prolcsvr, I.L.&O.R.. and Sch*>ol ot Computer and 
-RcMraich Assistant. I.P.AO.R., Syracuse University. 
hsiS. Rome Air Oevclopincnt Center, liritliss Air I 


Inlornution Science, S>racuse University, 
orce Base, Rome, Net^ t344l. 
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THE VIEWGRAPH MATERIALS 
of the 

A. GOEL PRESENTATION FOLLOW 
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OBJECTIVES 


• DEVELOP AN ANALYTICAL FRAMEWORK FOR ASSESSING HARDWARE- 
SOFTWARE PERFORMANCE AT A MACRO LEVEL. 

• EXPLORE TRADEOFFS BETWEEN PERFORMANCE MEASURES 

• DETERMINE OPTIMUM PARAMETRIC VALUE 

- WITHOUT CONSTRAINTS 

- WITH CONSTRAINTS 

• PROVIDE INSIGHTS INTO ROLES OF SYSTEM PARAMETERS IN DETERMIN 
ING PERFORMANCE 
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MARKOV MODEL FOR A HARDWARE-SOFTWARE SYSTEM 


• SYSTEM CONSISTS OF HARDWARE AND SOFTWARE COMPONENTS 

• THESE ARE SUBJECT TO RANDOM FAILURES 

• MAINTENANCE TIMES ARE RANDOM 
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A MARKOV MODEL 

• FAILURES. MAINTENANCE ASSUMED RANDOM WITH EXPONENTIAL 
DISTRIBUTIONS 

• KEY PARAMETERS: 

- S/W FAILURE RATE (Xj) 

- HAV FAILURE RATE (P) 

- S/W MAINTENANCE RATE (fij) 

- H/W REPAIR RATE (» 


A. God 
Syncuje t*. 
6of 25 



distribution of failure and maintenance times 

• Ti - TIME TO NEXT SOFTWARE FAILURE 

• Wj - SOFTWARE MAINTENANCE TIME 

• U - TIME TO NEXT HARDWARE FAILURE 

• V - HARDWARE MAINTENANCE TIME 
cdf of Tj = 1 - 

cdf of Wj = 1 - e-''** 

cdf of U = 1 - e-<5* 
cdf of V = 1 - e-'’'* 
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FAILURE RATES; 


• DETERMINE TIMES BETWEEN FAILURES 

• DEPEND ON: 

- CODE QUALITY 

- TESTING LEVEL 

- REDUNDANCY 

- FAULT TOLERANCE 

- WORK LOAD 
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MAINTENANCE 


INCLUDES ALL ACTIVITIES REQUIRED TO BRING THE SYSTEM BACK 
INTO OPERATION AFTER A FAILURE 

DOES NOT INCLUDE ENHANCEMENT OF CAPABILITIES 

DEPENDS ON 

- QUALITY OF PERSONNEL 

- S/W DEVELOPMENT METHODOLOGY 

- SYSTEM RECOVERABILITY 
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SOME USEFUL OPERATIONAL PERFORMANCE MEASURES 


• DISTRIBUTION OF TIME TO NEXT FAILURE 

• PROBABILITY OF SUCCESSFUL OPERATION FOR A SPECIFIED TIME 

• SYSTEM AVAILABILITY 

• DISTRIBUTION OF NUMBER OF FAILURES BY A SPECIFIED TIME 




CuniuldtivG Distribution Function of 
First Passage Tine to n. 
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Software 
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Surface of Average Availability vs. Repair Rates: Hardware- 
Softvare Systen (6 = .01, X = .05, t = 500). 
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OPERATIONAL COST MODELS 

• INCLUDE COSTS OF 

- FAILURES (HARDWARE OR SOFTWARE) 

- MAINTENANCE ACTIVITY AND PERSONNEL 

- SYSTEM DOWNTIME 

• PERFORMANCE MEASURES THAT AFFECT COST; 

- EXPECTED NUMBER OF FAILURES 

- SYSTEM UNAVAILABILITY 
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Software Repair Rate 












.05 


xpected Total Cost/Unit Tine vs. Repair Rates; Hardware- 
oft'/arc Systcn (G = .01, X ~ .05, t ~ 500). 
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OPTIMIZATION OF OPERATIONAL COST 


• FOR GIVEN COST FACTORS. EXPECTED TOTAL COST IS SOLELY DETER- 
MINED BY THE FAILURE AND MAINTENANCE RATES. 

• FOR A GIVEN SYSTEM, FAILURE OR MAINTENANCE RATES CAN BE 
OPTIMIZED. 
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OPTIMIZATION PROCEDURE 


• FUNCTIONS INVOLVED ARE NON-LINEAR 

• 2ND AND HIGHER ORDER DERIVATIVES ARE ALMOST IMPOSSIBLE TO 
OBTAIN 


• WE USE THE FOLLOWING METHODS: 


- POWELL'S SEARCH 

- VARIABLE METRIC METHOD AND 
DFP ALGORITHM 


} 


- LAGRANGIAN FUNCTIONS AS 
UNCONSTRAINED FOR USING VMM 


) 


UNCONSTRAINED 

CONSTRAINED 
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Expected Total Cost/Unit Tine vs. Repair Rate for 
Different Cost Factors (X = .05, t= 500). 
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Expected Total Cost/Unit Time (for Different 
Cost Factors) and Average Availability vs. 
Repair Rate (X = ,05, t = 500) , 
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CONCLUSIONS 

• MODEL PROVIDES AN ANALYTICAL FRAMEWORK FOR ASSESSMENT 

• NEEDS REFINEMENTS. RELAXATION OF ASSUMPTIONS 

• NEEDS VALIDATION 

• TOO MACRO? 

• FURTHER WORK ON OPTIMIZATION ALGORITHMS DESIRABLE 
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SOiTWARE RELIABILITY: 

A Comparison of Results Obtained from Established Software Reliability Models 

Martin H. Horn and William E. Thompson 
Columbia Research Corporation. Arlington, Virginia 
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ABSTRACT 

The software error detection process may be described as a stochastic process within the general 
Poisson family, but is distinguished by having rates which are changed by events, i.e. detection 
and correction of software errors. 

Two of the best known models of the software error detection process are here compared the 
Jelinski-Moranda model [I) and a Bayes inference model [2, 3], Simulation techniques are used 
to generate software-related system failure data which is analyzed by both models. Point esti- 
mates and confidence limits are compared. . 

It is demonstrated that uncertainty may be considerable for reasonable sample sizes and should 
certainly be considered in any application of these techniques. It is further demonstrated that 
the Jelinski-Moranda model is extremely sensitive to failure of data to follow the model’s inter- 
nal assumptions, often not providing any point estimates, a factor which may limit its usefulness 
in many real-world situations. The Bayes model is shown to be able to respond to the introduc- 
tion ot additional errors in the software correction process, a condition where error counting 
models such as the Jelinski-Moranda generally fail to converge. 


INTRODUCTION 

A maltunction or tailure of a computerized system is any response unacceptable to an ob- 
jective and consistent user. Each system failure falls within one of three categories: a hardware- 
related system failure (caused by such factors as error in hardware design or construction, or fail- 
ure ot component in service): a software-related system failure (caused bv error in desien’or 
implementation of computer programs, data or documentation) and the ambiguous or unknown 
tailure which cannot immediately be placed in either the hardware-related or'software-related 
categories. 

The overall reliability of the system may be defined as the probabilitv that the system will func- 
tion throughout a gi%en interval of time without a failure, given that ‘it was functioning at the 
start ot the interval. “Sottware reliability” is then defined as the probability that the system will 
function throughout a given time intervaJ-without experiencing a software-related system failure 
The mechanisms of hardware-related and software-related system failure are considered to be in- 
dependent point processes following a Poisson distribution with a consunt rate parameter. Tliis 
rate parameter, for software-related system failures, is denoted by X throughout this study and 
will be the parameter solved for in the reliabUity models. The reciprocal of X is tlie expected or 
mean time to the ne.\t software-related system failure, assuming no further correction of software 
errors, and thus may be related to “software reliability” as already dellned. 

In the development of a hardware-software system, software may be tested or debugged using 
test hardware which may not be the environment in which it eventually will operated In the test 
emironment, soitware errors are detected through exercise of the different program branches 
The discos cry of a software error in such a test process is defined, for the purposes of these re- 
liability models, as equivalent to a sottware-relaied tailure of the complete system in the field 


VI. Horn 
CRC 
I of 25 



THE MODELS 


The two models investigated in this paper estimate the rate parameter X in two different ways, 
both using as input the observed times between software-related system failures. The time to the 
fir^t software-related system failure is denoted as Xj ; the time between the first and second fail- 
ures is X 2 ; the time between the (n - l)th and nth failures is Xj^. The sum of all Xj, from i = 1 
to i = n, is the total elapsed time after n software-related system failures have occurred (Figure 
1 ). 

A. Jelinski-Moranda Model 


Jelinski and Moranda, in 1972, were among the first to offer a mathematical formulation for 
software reliability prediction. This model is one of a family of classical generalized Poisson mod- 
els. It assumes that a given software package contains a finite number N of errors initially. 

The fundamental assumption of the Jelinski-Moranda model is that X is proportional to the num- 
ber of errors currently present in the software, and that each error, once detected through a 
software-related system failure, is corrected immediately with no errors introduced in the correc- 
tion process. As each error is removed, X will decline in steps, but it is a constant rate parameter 
between each detection and removal of an error (Figure 2). 


The Jelinski-Moranda algorithm for estimating the value of N is: 


E 


1 

N-(i- 1) 


n X!Xi 




( 1 ) 


(n, N are positive integers with N > n) 


Once a best estimate for N is found, the proportionality constant 0 which represents one error’s 
contribution to the failure rate is computed as: 


n 

0 - ^ - C) 

The rate parameter X is then calculated from: 

X = 0lN-(n- 1)1 (3) 

The computer algorithm used in the solution of these equations first evaluates the left- and right- 
hand sides of equation (1) for N = n. It then evaluates these expressions for N = n + 1 , N = n + 
2, etc., establishing curves describing the left-hand and right-hand rides as functions of N. When 
an N is found where these curves intersect-or close to such an intersection if they do not inter- 
sect at an integer value of N-this N is the estimate of the number of errors originally presented 
and is used in calculation of 0 and X. 

Should .ieration of this procedure reach a preset maximum N without the cur\es intersecting, no 
further iteration is carried out and a message *‘NO SOLUTION FOUND” is printed. (In most 
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runs this maximum N was set at 200. In several runs, the maximum was increased to 400 with 
no significant change in the probability of not reaching a solution: it was concluded that in most 
cases when no solution is found the equation will not converge, rather than converge on an N 
higher than the limit.) 

The failure times xj were generated through a random number routine for input to the model to 
simulate a series of error detections. The Xj have an exponential distribution with a rate param- 
eter Xq = <>olNo - (i - 1)1 where the “true” values Nq and 4>o are input at the start of the run . 
and the Xj computed from: 

q = I -e-^oXjfNo -(i- D) (•») 

where tlie ^ is the random numbers uniformly distributed over [0. 1 1. 

After eacli “failure occurrence” the series of Xj so far generated would be used in a Jelinski- 
Moranda estimate of N and X. If the equation (1) did in fact convei^e, these estimates would be 
printed out. 

Fieure 3 represents a typical output from a Jelinski-Moranda run. The horizontal axis denotes n, 
the number of software-related system failures which have already occurred, while the vertical 
axis represents X as computed through the Jelinski-Moranda model at each successive n. The 
graph compares the X computed through the software development process to the “true" X„ de-- 
fined in terms of the input parameters and N„ by: 

= ^>o(No - (n - D) 

This follows from the fundamental assumption that X is dependent on the number of errors re- 
maining in the program, and that each error is corrected immediately upon discovery. Equation 
(4) may thus be restated as 

1 

,\„ = In (l-rn)-l (4) 

X„(n) 

.Althouch Xq is a step function which is constant between changes in n. it is shown in Figure 3 as 
a constantly declining function of n. 

There are caps in the craph of X as a function of n. as computed through the Jelinski-Moranda 
model. Where no X appears for a given n. Figure 3. no solution was computed for that n because 
equation (1) failed to converge. 

B. Bayes Model 

The second reliability model considered in this study is based on the Bayes inference procedure. 
The procedure assumes that X is a random variable and locates contidence limits of that variable 
from the posterior distribution iunction ot X. 

The Bayes inference procedure computes a posterior density function based on a previously com- 
puted prior density function and the experimental data obtained since the computation of the 
prior. At a given time T„ after k^ errors have been detected, the prior may be defined as: 
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[Reference 2) 


( 6 ) 


P(X) = 


(XTo)ko e*XTo 


roco + I) 

where T represents the gamma function (P(ko + 1) = kg! if kj, is an integer ^ 0). 


It is assumed here that the software in question does in fact contain further errors: that at a 
given time T' beyond the Tq the number k of errors already detected will increase to k' = k^ + 
k. At this time the posterior density function for X may be represented as: 


f(Xlk, T) 


(T' + To) (X(T' + 


r(k' i- ko + 1) 


(7) 


The distribution function is generated by numerical integration of this density function using 
Simpson’s rule. The density function is used as the prior in evaluation of the posterior density 
function for the next test interval where after another interval of time T' a number k' of errors 
is detected through software-related system failures. 

The distribution function, defined as: 

F(X'lk,T) = f f(Xlk, T)dX (8) 

o 

is used to compute upper and lower confidence limits and a medium value for X. The medium 
provides a point estimate for X, while the confidence limits are a measure of the uncertainty of 
this point estimate. For all the illustrations used in this study, upper and lower 80% confidence 
limits are employed-defining an 80% probability that the true X is between them. Integral (8) 
is evaluated for values of X' ranging from zero to one, generally in intervals of 0.01. Where the 
values of the integral is closest to 6.1, the upper limit of integration X' is printed out as the lower 
80% confidence Umit. The medium is that X’ where the integral value is closest to 0.5, and the 
upper 80% confidence limit that X' where the value of the integral is closest to 0.9 (Figure 4). 

Data is generated for the Bayes analysis by the same simulation process used for the Jelinski- 
Moranda analysis. The Bayes k and the Jelinski-Moranda n are the same, both equal to the num- 
ber of detected errors up to the Bayes T. which is the sum of the Jelinski-Moranda .\j. 

Figure 5 is a plot of a typical Bayes analysis using the model just described. As in Figure 3, the 
horizontal axis represents the number of errors already detected and the vertical axis represents 
the computed values of X. The curves, from the bottom, represent the lower confidence limit, 
median and upper confidence limit respectively. The true rate. X^, is shown. In this c-ase 
obeys the Jelinski-Moranda assumption of immediate error correction. The confidence limits 
conNcrge toward a given value as more errors are discovered, narrowing the range in which the 
true X is likely to be found. 


PROCF.DL'RE FOR COMP.ARISON MODELS 

For each comparison between these models, a series of “failure times” was generated and stored 
on tape so that the identical data could be analyzed using both models. 
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Figure 6 is an overlay of Figures 3 and 5: a direct comparison of the Jelinski-Moranda and 
Bayes model estimates of X for the same input data. In this case the data was generated using 
Equation (4): the Jelinski-Moranda assumption' of immediate error correction holds. As might 
be expected, the more data is available the closer both models estimate to the “true” X^. 

However, as n approaches 100 (the input Nq) the Bayes model converges to approximately half- 
way between the original X^ and zero (the theoretical X when all errors have been corrected). 

Figure 7 is a similar run with different input data. Again, the Xj are computed using the Jelinski- 
Moranda hypothesis of immediate error correction; again the Bayes model converges to a value 
higher than X^ as errors are corrected; again there is a considerable range of n where no Jelinski- 
Moranda estimate is avaUable. Comparison of Figures 6 and 7 shows that the Jelinski-Moranda 
wlue docs not appear where X as indicated by the Bayes model is an increasing function of n. 
This corresponds to an increase in the rate of error discovery per unit time-directly contradicting 
the Jelinski-Moranda hypothesis of decreasing X as errors are corrected. Figure 7 also plots the 
“arithmetic mean” defined as n/T or failures per unit time. This curve closely follows the Bayes 
median. 


MODEL COMPARISON FOR VARIOUS INPUT HYPOTHESIS 

Figure 8 is another run in which the Jelinski-Moranda hypothesis was followed in generation of 
input data. The first “failure” occurred at an abnormally long time after start of testing, as com- 
pared to Xq. As in all other runs, the time to the first failure is used to compute the first Bayes 
prior, and thus the prior X is abnormally low. As further “failures” occurred, the rate X was ob- 
served as increasing since it had been low at the start. The Bayes model followed this observed 
increase and converged toward the given X^,. However no solution was obtained from the 
Jelinski-Moranda model at any point of this run. This is further indication that the Jelinski- 
Moranda model does not provide any estimate of X when the trend of the jate is upward. 

In all the examples presented so far, the input to the simulation process has conformed to the 
Jelinski-Moranda assumption that a software error is corrected immediately upon detection. How- 
ever. in the “real world” this may not always be true; an attempted correction may not remove 
an error, or may introduce a new error. The simulation process was therefore modified to allow 
for this possibility. 

In Figure 9 is shown the result of a run in which, upon detection of a software error, a 75% 
probability was assumed that the error would be corrected. The other 25% of the time the cor- 
rection attempt was unsuccessful: the number of errors remaining in the program was unchanged. 

To provide this hypothesis, a random number was generated; if this number was less than or 
equal to 0.75 the number n' of errors corrected would decrease by 1, othenvise n' would remain 
unchanged. The failure rate in Equations (4) and (5) would then be dependent on n', the 
number of errors actually corrected, which may now be different from n (or i), the number of 
software-related system failures observed (Figure 10). 

\) = - (n' - 1)) ( 9 ) 


No solution was found anywhere on this run by the Jelinski-Moranda model, 
made using this hypothesis, no Jelinski-.Moranda solutions were found except 


( 10 ) 

In the several runs 
in those cases where 
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the immediate correction hypothesis was obeyed for the first several iterations (n' - n); the 
Jelinski-Moranda model would produce an estimate of X for these iterations but not after the 
first instance of non-correction. Once the model stopped producing estimates, it would not re- 
sume at any time. The Bayes model is shown to converge-and since the decrease of X^ is slower 
than before*, the Bayes model here used, which searches for a constant failure rate, converges 
"better" to than it did when the Jelinski-Moranda hypothesis was followed. 

Figure 1 1 shows a nm similar to Figure 9. but in which the probability existed of the introduc- 
tion of further errors during the software correetion process. A random number determined 
whether n’ as defined above decreased (error corrected), increased (error not corrected arid new 
error introduced) or remained the same Terror not eorreeted. or corrected but new cnoT intro- 
duced) (Fieure 12). luiuations (*?) and (10) were again used, once an n was established, to gen- 
erate "failure times". The result is the same as the previous run; the Jelinski-Moranda model is 
usable to handle this deviation from its internal assumptions and does not reach a solution 
anywhere. 

In another tested hypothesis, correction of errors follows the Jelinski-Moranda assumption tor 
the first fifteen errors detected; each is corrected immediately. The next five errors are not 
corr-'ted- five software-related svstem failures oecur and no successful error airrection is made. 
After'these five failures, a correction is finally achieved and the process again follows the Jelinski- 
Moranda assumption. In Figure 13. X„ is shown to decline from n = 1 to n - 15_. From n - l.v 
to n = ■'0 \ remains constant (the number of errors corrected, n . remains at 1> while the num- 
ber of failures, n. increases). After n = 20. declines again, at the same rate as before ii - 1? 
as all errors are again corrected immediately. The Jelinski-Moranda model produces estimates of 
X up until n = 15- the point at which its hypothesis is violated. Thereafter, it does not con- 
verge to a scMut ion even after the region of non-correction has been left. Again the Bayes model 
is shown as better able to follow a ehanging failure rate, even though it does not conform to the 
Bayes internal hypothesis of a constant rate. 

Figure 14 is a result of using a hvpothesis similar to that of Figure 13. but with additional errors 
int'rvsJua-d. From n = 1 to'n = 20. the Jelinski-Moranda hypothesis is followed. From n - 1 

to u = 25. one new error is introduced at each failure; as n increased by 1. n decreases by 1. 
After n = '^ the Jelinski-Moranda hypothesis is again followed. Tlie graph shows X„ decreasing 
until n = 2oVthen increasing until n reaches 25. after which it decreases again at the same rate as 
before. .As in Figure 13. the Jelinski-Moranda model produced estimates of X until the point at 
which its hvpothesis was violated, and did not converge again even once its hypothesis was again 

valid. 

In F**’ure 15. an opposite perturbation was applied to the Jelinski-Moranda hypothesis. .A cor- 
rection was assumed to follow each error detection; however it was jxjssible to correct more than 
one ‘ran at a time. (.A second error may he discovered in examination of the code in the pro- 
cess of avrrection of the original error which caused a failure.) If one error was coaected. n 
increases bv 1; if two errors were corrected at one time, n' increases by 2 as n increases by 1. 
Here the Jelinski-Moranda model was able to reach estimates of X although they appear to be 
iiniformh low; the Bayes model converges to a high value. 


ASSFRTIONS RliSULTlNC. FROM COMPARISON OF MODl.LS 

It h*N be*n ^hown that the Jelinski-Moranda model will provide an estimate of X when the as- 
sumptions underlvmg the model are valid, but any ivrUirbation of these assumptions which has 
the effect of increasing the rate X^ above that otherwise encountered will cause this model not 
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to converge to an estimate. On the other hand, the Bayes constant-rate model which is com- 
pared to it will always provide a point estimate and confidence limits, but may converge to a 
value above as n increases. 

Even where the Jelinski-Moranda hypothesis holds, but the actual rate of error detection is shown 
to (temporarily) increase, the Jelinski-Moranda model will not reach a solution (Figures 6, 7, 8). 
The assertion suggests itself that this non-convergence of the Jelinski-Moranda model may be in- 
herent in the definition of the model; that it may be proven, from the definition of the model, 
that it will not converge if the trend in the Xj is decreasing, indicating that failure occurrence 
frequency is increasing. 

Sukert, et al. [5] have demonstrated that the maximum likelihood estimator of the Jelinski- 
Moranda N does not exist in the case where the time interval Xj between error detections is con- 
stant or decreasing as n increases, a finding which agrees with the assertions made here that no 
Jelinski-Moranda solution is found when the trend of the \y is decreasing (corresponding to an 
increase in rate of error detection). 


CONCLUSIONS: FURTHER DEVELOPMENT 

The Bayes model, even in the current constant-rale assumption, has been shown able to respond 
to perturbations in error discovery rates which cause the Jelinski-Moranda model to fail to reach 
an estimate of X. 

The Jelinski-Moranda model is one of several error count models, some of which assume an in- 
creasing rate and some a decreasing. Moranda (61 has discussed the application of several of 
these models, but none of them is suitable as a single reliability model which will follow any 
change in rate of failure occurrence or error detection. Littlewood (31 and Goel [41 have de- 
veloped more flexible Bayes models, allowing increasing and decreasing X^. Development is con- 
tinuing at CRC on modifications to the Bayes model which will permit it to follow a declining 
rate trend if there is in fact such a trend. Such a model would recognize a trend and modify the 
prior in accordance with the trend, so as not to be influenced by extreme tluctuation of rate 
early in the software development process, although in no way would it stop production of es- 
timates should the trend be reversed. 
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OCCURENCES OF SOFTWARE RELATED SYSTEM FAILURES 


FU;UIU: 1; Definition of n and 
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JELINSKI-MORANDA MODEL ASSUMPTIONS 


1) EACH ERROR IS CORRECTED IMMEDIATELY UPON DETECTION. NO NEW 
ERROR INTRODUCED. 


2 ) 


A IS PROPORTIONAL TO NUMBER OF 
A DECLINES BY CONSTANT AMOUNT 


ERRORS REMAINING IN SOFTWARE, 
cl) AFTER EACH ERROR IS CORRECTED. 


3) A IS CONSTANT BETWEEN DETECTIONS OF ERRORS. 


A 

N<l> 

(N-U^l* 

(N-2)‘l» 

(N-3)0> 



X, 

A 


n=2 



X, 

A 


n=3 




l 


n=4 



1 
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RESULT OF JELINSKI— MORANDA ESTIMATE 

No=1 00 Oo=.001 



NUMBER OF OBSERVED 
SOFTWARE— RELATED 
SYSTEM FAILURE 


1 
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RESULT OF BAYES ESTIMATE 
N.-100 ‘K=.001 

A, FOLLOWS JELINSKI MORANDA HYPOTHESIS OF 
IMMEDIATE ERROR CORRECTION. 






NUMBER OF OBSERVED 
SOFTWARE-RELATED 
SYSTEM FAILURES 


1 



COMPARISON OF BAYES AND 
JELINSKI—MORANDA 


No=100 00=001 

Ao FOLLOWS JELINSKI—MORANDA HYPOTHESIS OF 
IMMEDIATE ERROR CORRECTION. 
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N„= 1 50 
d ) =.001 

1 FOLLO 

Aq 


A. 




No=100 

Oo=.001 

ERRORS UNCORRECTED AT RANDOM 
(25% PROBABILITY OF NO CORRECTION) 
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NOT ALL ERRORS CORRECTED UPON DETECTION 

JELINSKI-MORANDA HYPOTHESIS HOLDS ONLY WHEN AN ERROR IS IN FACT 
CORRECTED. 

A IS NOW PROPORTIONAL TO NUMBER OF ERRORS REMAINING-WHICH MAY 
NOT BE EQUAL TO N-n. 



) 


PRIOR 


A 



0 


T NEW 


No=150 

ct)o=.001 


NEW ERRORS INTRODUCED AT RANDOM 



(NO MORANDA SOL’N) 


R 

IDUCED KTCUIUi 1 1 
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1 NEW ERROR MAY BE INTRODUCED IN CORRECTION PROCESS 

INTRODUCTION OF NEW ERROR RAISES A BY A FACTOR 


r 

1N«I» 

(Njl)<l> 

(N-J)<l> 



i NEW ERROR 

INTRODUCED 


X. 

A 


\ 

1 


n=5 




TIME 

FIClJUl! 12: Hypothesis used In f'cneratlon of MRure 11 
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No=1 50 
tI>o=.001 

NEW ERRORS INTRODUCED n=20 TO n=25 
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MEASURING THE DEVELOPMENT PROCESS 
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MEASURING THE DEVELOPMENT PROCESS: 
Symbology and Spatial Arrangement: 

Effects on the Comprehension of Software Specifications 

S. B. Sheppard, E. Kruesi, and B. Curtis 
ITT 


The costs and reliability of software development and maintenance are affected by. how efficiently 
programmers can perform their tasks throu^out the software life-cycle. Easily understood speci- 
fications are a primary criterion for a successful sv-stem development effort. This experiment in- 
vestigated the understandability of various specification formats in a laboratory environment. 

Two dimensions of specification formats were examined; the type of symbology and the spatial 
arrangement of the information. 

Three types of symbology were evaluated: natural language, constrained language and ideograms. 
The natural language consisted of a description of a program in English narrative. The constrained 
language was a modified version of PDL (Program Design Language). The ideograms were com- 
prised of the standard IBM flow chart symbols. 

Three spatial arrangements were also evaluated: sequential, branching and hierarchical. In the 
sequential versions, both the flow of control and the nesting levels were arranged vertically. (Tne 
sequential arrangement is the standard format for presenting PDL and English narrative.) The 
branching arrangement was similar to a structured fiow chart and had a vertical flow of control 
and horizontal nesting. The heirarchical arrangement was somewhat tree-like in nature with a 
horizontal flow of control and vertical nesting. 

Each of the three types of symbology were presented in three spatial arrangements, resulting in 
nine specification formats, these nine formats were prepared for each of three small programs 
(approximately 50 lines of code). 

Seventy-two participants were asked to study a set of specifications and then answer a series of 
questions from the specifications. The questions were presented interactively on a (TRT. The 
answers and response times were recorded automatically. Each participant saw specifications for 
ihe three different programs. Across the three programs, each spatial arrangement and each type 
of symbology was seen once. 

Ekhteen questions comprised each experimental task. These included; 5 fonvard-tracing ques- 
tions. 5 backward-tracing questions and 8 input-output questions. For the forward-tracing ques- 
tions! the participants were given a set of conditions from the specifications. Their task was to 
trace through the specifications and find the first statement executed under those conditions. 

For the backward-tracing questions, they were presented with a statement. Their task was to 
determine the relevant conditions which held when that statement was executed. For the input- 
output questions, they were some data and were required to give the output values. 

The level of accuracy was high (89^) and did not differ across the nine formats. Both forward- 
and backward-tracing questions were answered more quickly from specifications presented in con- 
strained language or Ideograms than in natural language. Forward-tracing questions were answered 
most quicklv from a branching arrangement, and backward-tracing questions were answered more 
quickly from branching and hierarchical arrangements. Response times to the mput-output ques- 
tions did not varv significantly as a function of the type of symbology or the spatial arrangement. 
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When asked which type of symbology they preferred, the majonty of the participants preferred 
the constrained language, while the fewest preferred natural language. In terms of the spatial ar- 
rangement, the branching airangement was the most preferred while the hierarchical arrangement 

was the least preferred. 


These results extend previous research on presentation formats by demonstrating the separate con- 
tributions of symbology and spatial arrangement to comprehension. 
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• VARY THE TYPE OF SYMBOLS 

IDEOGRAMS 

CONSTRAINED LANGUAGE 
NATURAL LANGUAGE 

t VARY THE SPATIAL ARRANGEMENT 

- ^ BRANCHING 
SEQUENTIAL 
HIERARCHICAL 

• RESULT IS NINE DIFFERENT FORMATS 
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WHICH IS better: 

program design language (pdl) or flowcharts? 

o c 
-n & : 

ONE approach: 

COMPARE THESE TWO REPRESENTATIONS ON ONE OR 
more tasks (E.G., COMPREHENSION, CODING) 

s ? 

THIS HILL TELL US 

WHICH IS BETTER BUT NOT WHYi 

r- t 


ARE DIFFERENCES DUE TO; 

^ - TYPE OF SYMBOLS (CONSTRAINED UN6UAGE VS IDEOGRAMS) 

- SPATIAL ARRANGEMENT (SEQUENTIAL VS BRANCHING) 


BRANCHING 

IDEOGRAMS 














?AUc I3 

OF poo:^ QUALITY 


PROGRAM TO SIMULATE 
WAITING TIME AT A 
GAS STATION 

Set the initial values 

FOR THE random NUMBER 
CENERATOR^TO ZERO. 

Read the total number 
OF minutes FOR THE 

SIMULATION FROM THE 

FILE MNFjlE'. 

Do THE STEPS TO THE 
RIGHT while THE NUMBER 
OF MINUTES FOR THE 
SIMULATION IS LARGER" 
THAN ZERO. 


BRANCHING 

NATURAL 

LANGUAGE 


Set the following variables to zero: 

THE MAXIMUM WAITING TIME. THE ACCUMULATED 
WAITING TIME, AND THE NUMBER OF CUSTOMERS. 
Set THE SIMULATION TIME TO ONE MINUTE. 


Do THE STEPS TO THE RIGHT WHILE THE 
SIMULATION TIME IS LESS THAN OR EQUAL - 
TO the NUMBER OF MINUTES FOR _T»TE 
SIMULATION. 


Then 


If the maximum waiting time is greater 

THAN ZERO 

I 

Otherwise 


Decrease it by one. 

I 


If the numbek returned by the random 

number GENERATOR IS GREATER THAN OR 

equal to 0.9 


Then 


Otherwise 


Increase the number of customers by one, 
INCREASE the ACCUMULATED TOTAL WAITING TIME 
BY THE MAXIMUM WAITING TIME, AND INCREASE 
THE MAXIMUM WAITING TIME BY EIGHT MINUTES. 

I ^ 


Increase the sihulation time by one minute. 


Calculate the average waiting time by 
DIVIDING the accumulated WAITING TIME 
BY THE number of CUSTOMERS. 

Print the average waiting time. 

I 

Read the total number of minutes for 

THE NEXT SIMULATION FROM Tm£ FILE 

'INFIIE^ I 


This completes the process necessary 
TO SIMULATE waiting TIME AT A GAS 
STATION. 


S. Sheppard 
GE 

9of23 



futdddqs *S 


) 






S. Sheppaxd 


) 


) 




!U JMiijAII 
I in Al A - 


'.I I tP«l INI I lAl VA| III 
»U« iMl NKMf'^N 


t^l AM IMI lUlAI NlHIil N |)il nil Mil*'. H|MM Hil||t |H| 
-ll NJNlJII'i lUM iMt — NUMblN Ut MINUTl^; f UW Ma 


Imp. niMHitut. TM| 

-TO simulate haiting time at a gas 



.11 I**| .IMtHAIION TIME TO '.fil M1NUTT. 


|j rn( MArfMUM *«A| r IN'I 
T I *<l I >1 A Tf B TliAfl 



MlUUlfS FOB TMt simulation. 


time by the number of 



SIMULATION FROM 

THE file ‘INFILE* 



If the number returned B/ the random 

NUMhTR GENERATOR IS GREATER THAN OR 
lUIJAl TO 0.9 


.Increase the simulation 
’ TiMi Bv one minute. 


Ihi n 


1 >«> K I A ' I 1 k| NiIMI.I rf III 
( U'. T'/MfR '. B/ ONE . F£>' >' 

|H( AM IJHUI Airt) IMTAl 
NAIIING T 1 M[, AND IfirREA'.E 
Till MA/IHljH WAITING TIME 
lu E I', Hi MINUIF S. 


Otherwise 


HIERARCHICAL 

NATURAL 

LANGUAGE 


ORIGINAL PAGE 13 
OF POOR QUALITY 






PROGRAM GAS STATION 


SEQUENTIAL 

PDL 


SET I - 0 
SET J = 0 

READ FROM 'INFILE': N 

DO WHILE N>0 

SET SERVE = 0 
SET WAIT = 0 
SET COST « 0 
SET TIME * 1 

DO WHILE TIME^N 

IF SERVE>0 

THEN 

SET SERVE = SERVE - 1 
ENDIF 

IF RAN (IJ)i0.9 
THEN 

SET COST » CUST ♦ 1 
SET WAIT - WAIT ♦ SERVE 
SET SERVE * SERVE + 8 

ENDIP 

SET TIME = TIME ♦ 1 

ENDDO 

SET AVWAIT * WAIT/CUST 
PRINT 'AVERAGE WAIT IS', AWAIT 
READ FROM 'INFILE': N 

ENDDO 

END OF GAS STATION 
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PROGRAM TO SIMULATE WAITING TIME AT A GAS STATION 

Set the initial values for the random number generator to zero. 

Read the number of minutes for the simulation from the file 'INFILE'. 
Do STEPS 1 through 5 WHILE THE NUMBER OF MINUTES FOR THE SIMULATION 
IS LARGER THAN ZERO. 

1. Set THE FOLLOWING VARIABLES TO ZERO: THE MAXIMUM 

WAITING TIME/ THE ACCUMULATED WAITING TIME/ AND 
THE NUMBER OF CUSTOMERS. 

2. Do STEPS A THROUGH C WHILE THE SiWlATION TIME IS 
LESS THAN OR EQUAL TO THE NUMBER OF MINUTES FOR THE 
SIMULATION. 

(A) If the maximum waiting time is greater than zerO/ 

DECREASE IT BY ONE. 

(B) If the number returned by the random number 

GENERATOR IS GREATER THAN OR EQUAL TO 0.9/ INCREASE 
THE NUMBER OF CUSTOMERS BY ONE/ INCREASE THE 
ACCUMULATED WAITING TIME BY THE MAXIMUM WAITING TIME/ 
AND INCREASE THE MAXIMUM WAITING TIME BY 8 MINUTES. 

(C) Increase the simulation time by one minute. 

3. Calculate the average waiting time by dividing the 

accumulated WAITING TIME BY THE NUMBER OF CUSTOMERS. 

Print the average-waiting time. 

5. Read the total number of minutes for the next simulation 
FROM THE FILE 'INFILE' . 

This completes the process necessary to simulate waitiwg time at a 

GAS STATION. 
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Site 
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ANSWER QUESTIONS ON-LINE WHILE STUDYING DOCUMENTATION 
72 PROFESSIONAL PROGRAMMERS AVERAGING 7 YEARS EXPERIENCE 


SCIENTIFIC (rocket TRAJECTORY) 

DATA BASE ( STORE INVENTORY) 

scientific/data base (airport scheduling/simulation) 


ACCURACY OF RESPONSE 
RESPONSE TIME 
PREFERENCES 
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• Forward Tracing 


• Backward Tracing 


• Input-Output 



o 
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Forward Tracing Questions 



NATURAL 

LANGUAGE 

CONSTRAINED 

LANGUAGE 

IDEOGRAMS 

Total 

sequential 

WM 

32.9 

37.8 

NO.l 

BRANCHING 

■■ 

30.4 

36.0 

37.1 

HIERARCHICAL 


41.9 

38.9 

m.N 

Total 

45.9 

35.1 

37.6 

39.5 


Note. Individual cell means represent 120 responses 
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Backward Tracing Questions 



Note: Individual cell means represent 120 responses 
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Input-Output Questions 



NATURAL 

LANGUAGE 

CONSTRAINED 

LANGUAGE 

IDEOGRAMS 

Total 

SEQUENTIAL 

43. A 

41.9 

38.9 

41.4 

BRANCHING 

43.3 

36.2 

44.3 

41.4 

HIERARCHICAL 

41.9 

44.8 

34.9 

40.5 

Total 

42.9 

41.0 

39.4 

41.1 


Note: Individual cell means represent 192 responses 
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TYPE OF SYMBOLOGY: 


CONSTRAINED LANGUAGE (PDL) 

53X 

IDEOGRAMS 

33T 

NATURAL LANGUAGE 

I'lT 


lOOZ 


SPATIAL ARRANGEMENT: 


BRANCHING 

t<87. 

SEQUENTIAL 

3G7. 

HIERARCHICAL 

1G7 


1007 
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RELATIONSHIP OF EXPERIENCE 
TO PERFORMANCE 
(Average Response Time) 


INFORMATION SYSTEMS 
PROGRAMS 


SOFTWARE MANAGEMENT 
RESEARCH 


Years Programming 
Number of Languages 


+.23* 

-.A9** 



*p<.05 

**P<.01 
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• For a comprehension task^ natural language is 

NOT AS GOOD AS CONSTRAINED LANGUAGE OR IDEOGRAMS 

• The branching arrangement with succinct symbols 
(constrained lang'uage or ideograms) is best for 


comprehension. 



MEASURING THE DEVELOPMENT PROCESS: 
Software Design Coupling and Strength Matrices 

R. D. Cruickshank and J. E. Gaffney, Jr. 

IBM Corporation, Federal Systems Division, Manassas, VA 


OVERVIEW 

Meyers' has defined two concepts of considerable importance to representing the degree of 
‘goodness’ of a software design, ‘coupling’ and ‘strength’. ‘Coupling indicates t^he degree of in- 
terconnectedness of modules. ‘Strength’ indicates the degree of cohesiveness of module function. 
Both of these measures relate to the ease of modification of a module. If modules have relaUvely 
low strLgth, then the function they effect wUl be relatively diffused, and changes m a function 
wUl be propagated among a plurality of modules at decreased efficiency and increased cost. 

Cruickshank and Gaffney^ have developed a set of metrics relating to the ‘goodness’ of a “f'ware 
design and its degree of completion. A (software) metnc is a math^atical measure of a ^dy ot 
iftware that is sensitive to differences in software characterbtics. Two of these metrics, for 
Suphng’ and ‘strength’, are described in detaU here. The effect of these metrics ,s to transform 
tSse qualitative concepts into quantitative measures. These metrics are a tool for evaluating a 
design\nd comparing it with alternatives. These measures have been applied to a design written 
in a design language. Process Design Language (PDL). 


STRENGTH 


Meyers has defined two types of high strength modules. The first, ‘functional strength\ refers to 
a Jodule in which all of the elements relate to the performance of a single function. The other, 
‘informational strength’, relates to a module that performs multiple functions with a single data 
structure The strength metric recognizes that ‘strength’ is a two-dimensional concept, in which 
the relative numbers of (unique) inputs and outputs are represented and the depth of Pro^e^mg 
(number of actions indicated) is included. This is necessary in ord^er o cover the ^ 

in which modules exhibit high functional or informational strength. ^ 
outputs/no. of module inputs and X = 1 /total number of assignment. RUN ^nd CALL PDL_ 
statements. Tlien S = X + jY, a complex number. For comparative purposes, S the strength 

/ v 2 + \2 As an illustrative example, a module exhibiting higher informational strength might 
fake from types of data A, B. C, and D and do many things to them. As an example take the 
following code sequence which would also appear the same in the design language, PDL. 

0 ,:-A + B + C + D 


O 2 : = A2 +B2 +C2 +D2 
O 3 : = (A/B)MC + 3D1 

In this case. X = 1 / 3 ; Y = 3/4 = 0.75; and S = 0.819. A module of lower informational strength 
could be one that performs several unrelated functions: 


0,; = A + B 
Oj; = C + D 
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Here, X = 1/2: Y = 2/4 = 0.5; and S = 0.707. 

Note the relationship between the strength measures and the transfer function of an electronic 
circuit, which has a gain (rate of output/input magnitudes) and a phase (measure of delay through 
a circuit). 


COUPLING 

Coupling is a measure of the relationships among modules, whereas strength is a measure of rela- 
tionships among the elements within a module. Coupling measures the degree to which two mod 
ules are interconnected. The less the coupling, the better the design, because intellectual control 
of design is facilitated and the propagation of changes is minimized for a given module. Let Z 
equal the average number of input and output items shared by this module and another one. 

This figure can be normalized, if desired, to vary between 0 (for high coupling) to 1 (for low 
coupling). If k = 1 - e.xp (-0.66943/Z); then if Z = 1, k = 0.988; and if Z = 0.25, k = 0.93. 


OBSERVATIONS 

Software metrics create a new dimension for the evaluation of software. They can greatly im- 
prove the manageability of the software development process and enhance the growth of the re- 
sultant product. The set of metrics should include ones for coupling and strength to replace the 
qualitative measures for these items used heretofore. The data for the mathematical measures of 
coupling and strength given here can be derived automatically if a system, such as PSL/PSA, de- 
veloped by the University of Michigan, is employed to support the design process. Very import- 
antly, these metrics provide a method of comparing designs and suggesting improvements to soft- 
ware designs. 
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OBJECTIVE OF OUR WORK IS TO QUANTIFY SOME MEASURES OF 
GOODNESS/BADNESS OF DESIGN 
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TWO IMPORTANT CONCEPTS* REPRESENTING THE DEGREE OF 
GOODNESS OF A SOFTWARE DESIGN ARE: 


STRENGTH - INDICATES COHESIVENESS OF MODULE FUNCTION 

COUPLING - INDICATES DEGREE OF INTERCONNECTEDNESS OF 
MODULES 


•SUGGESTED BY MYERS IN "RELIABLE SOFTWARE THROUGH COMPOSITE 
DESIGN" {PETROCELLI/CHARTER. 1975) 
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WE PROPOSE QUANTIFYING THESE CONCEPTS SO THAT 
ALTERNATIVE DESIGNS CAN BE COMPARED 



STRENGTH METRIC 


TWO DIMENSIONAL, REPRESENTS: 

• RELATIVE NUMBER OF INPUTS AND OUTPUTS 

• DEPTH OF PROCESSING 


X 



Y = NO. OF UNIQUE MODULE OUTPUTS 
NO. OF UNIQUE MODULE INPUTS 

(Y ANALOGOUS TO "GAIN" IN A CIRCUIT) 

X = 1/NO. OF ASSIGNMENT STATEMENTS 

(1/X ANALOGOUS TO "PHASESHIFT" IN A CIRCUIT) 

S = Xo + JYo 

S = STRENGTH = + Y2 
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TWO TYPES OF HIGH STRENGTH MODUl ES - 

FUNCTIONAL - ALL OF THE ELEMENTS RELATE TO THE PERFORMANCE 
OF A SINGLE FUNCTION 

INFORMATIONAL - MULTIPLE FUNCTIONS PERFORMED WITH A SINGLE 
DATA STRUCTURE 
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EXAMPLE OF LOWER FUNCTIONAL STRENGTH 
R: = y/ A + B 
Z: = C + D 
0: = R + Z 
X = 1/3; Y = 1/4 
S = y (1/3)2 +(1/4)2 = 0.414 

EXAMPLE OF HIGHER FUNCTIONAL STRENGTH 
0; = A + B + C + D 
X = 1/1; Y = 1/4 
S = y(1/1)2+ (1/4^2 = 1.0308 
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EXAMPLE OF HIGHER INFORMATIONAL STRENGTH 
Oi*. = A + B + C + D 
Oj: = a2 + b2 + C2 + D2 
O 3 : = (A/B) X C + 3D 
X = 1/3; Y = 3/4 
S = y (1/3)2 + ( 3 / 4)2 = 0.81939 

EXAMPLE OF LOWER INFORMATIONAL STRENGTH 
0,; = A + B 
O 2 : = C + D 
X = 1/2; Y = 2/4 
S = y (1/2)2 + (2/4)2 = 0.707 
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SOME STRENGTH DATA 


PROCEDURE 

UNK 

INPUTS (I) 

m 

OUTPUTS (0) 

Y=0/I 

NO. OF 

ASSIGNS (1/X) 

X 

S=Jx^ + Y? 

1 

5 

A 

.8000 

].l 

.0083 

.8051 

2 

6 

5 

.8333 

9 

.1111 

.8A07 

3 

7 

5 

1 

.71A3 

10 

.1000 

.7213 


5 

A 

.8000 

5 

.2000 

.82A6 

5 , 

A 

5 

1.2500 

7 

.1A29 

1.2581 







COUPLING METRIC 

Z = REPRESENTS AVERAGE OF INPUT AND OUTPUT ITEMS SHARED BY 
THIS MODULE AND ANOTHER ONE. 


EXAMPLE: FOR MODULE MX 



NUMBER OF ITEMS: 

1 FROM M, 

2 FROM M 2 
1 TO M 4 
1 TO Mg 

3 TO Mg 

Z = 1+2 + 1 + 1+ 3 = ^ = 1-6 
5 5 

CAN COMPUTE AN AVERAGE Z OVER A SET OF MODULES 

EXAMPLE: Z = 1.6 + 2.0 -i- 3,0 = 2.2 FOR A 3 MODULE PROGRAM 

3 
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NUMERICAL STRENGTH AND COUPLING METRICS PROVIDE A 
MEANS FOR QUANTITIVELY COMPARING ALTERNATIVE DESIGNS 
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MEASURING THE DEVELOPMENT PROCESS: 
A Tool for Software Design Evaluation 

Susan S. Moy 
Logicon, Inc. 


ABSTRACT 

This paper described the Design Metrics Evaluator (DME), a component of an automated software 
design analysis system developed by Logicon. The DME quantitatively evaluates software design 
attributes. Its use directs the analyst’s attention to areas of a procedure, module, or complete 
program having a high potential for error. 


INTRODUCTION 

As the usage of computers has increased and their areas of application have broadened, software 
reliability has become an issue of major concern. Considerable effort has been devoted to assess- 
ing software reliability, and studies of software quality characteristics and ^ftware evaluation 
technologies have produced reliability models and complexity metrics based on various approaches 
and parameters. Nevertheless, there are still severe limitations on our ability to evaluate the qual- 
ity of computer software quantitatively and automatically. 

At Logicon, efforts to enhance software evaluation methodology have produced a system of soft- 
ware design analysis tools called LOGICFLOW. LOGICFLOW has the following components: 

• A simple design language sufficiently versatile to relate high- and low-Ievel, structured and 
unstructured designs 

• A design language processor to detect logical errors 

• A translator to translate the design language into FORTRAN or JOVIAL 

• A Howcharter to generate Oowcharts from design language, FORTRAN, and several as- 
sembly languages 

• A cross-reference generator to create an index of all design tokens (or symbols) 

• A design metrics evaluator to generate metrics that reflect the characteristics of a software 
design 

The Design Metrics Evaluator provides a quantitative measure of design quality. It facilitates en- 
forcement of software development standards, encourages better design by providing feedback on 
design quality during the design phase, and directs attention to areas where there is a high poten- 
tial for errors. Tliis early detection of errors in ti e design phase can reduce software production 
cost, improve software reliability, and contribute to timely completion of the software. 


METHODOLOGY 

The quality of a software design can be evaluated by comparing specific design characteristics with 
predetermined evaluation criteria. The approach used to develop the Design Metrics Evaluator was 
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to identify desirable software design characteristics, determine how to measure them, and estab- 
lish a set of evaluation criteria. 

The first step in developing the Design Metrics Evaluator was to identify a set of nonoverlapping, 
desirable attributes of software design. The pertinent literature was surveyed and all relevant at- 
tributes were extracted and recorded. Table 1 presents the list of desirable attributes compiled in 
this way. All members of this initial set were examined, synonymous or redundant attributes 
were grouped together, and the most meaningful or inclusive term in each group was chosen as 
the representative “name” for that group. The resulting attributes were then organized hierarchi- 
cally, with attributes identified as goals at the top level and contributing factors at the lower levels. 

The attributes that emerged as goals were usability, maintainability, and generality. Usable soft- 
ware is easy to work with and produces reliable results. 

Maintainable software is understandable, modifiable, and verifiable. General software adapts 
readily to new conditions and different configurations. Reliability and human factors are support- 
ing factors for the goal of usability; understandability, verifiability, and modifiability support 
maintainability: and reusability and transferability support generality. These supporting factors 
can be further broken into components as shown in Figure 1. 

Once this basic hierarchy had been formulated, the definitions of the design attributes were ex- 
amined and a number of design criteria were identified for each. For example, some of the cri- 
teria associated with structuredness are: Is the design modularized in accordance with the major 
system functions? Are the modules organized heirarchically? Are all modules separate and 
distinct? 


After the desirable software design attributes had been identified, design language constructs were 
studied and measurable features such as program length, statement size, etc., were identified. A 
prototype Design Metrics Evaluator was developed to collect statistics automatically on the design 
language constructs. These statistics were formulated into an initial set of metrics. Analysis of^ 
these metrics and their relationship to the design attributes revealed that not all design character- 
istics identified earlier are quantitatively measurable.* Tlie metrics collected reflect the fact that 
structural characteristics (such as the number of unconditional branches in a procedure) are the 
most directly measurable. Difficult to quantify are the design characteristics related to the func- 
tional aspects of a design (for example, the accuracy of the algorithms used). 

The initial set of design metrics was found to have the following limitations: 

• Knowledge of the evaluation criteria is required to interpret the metrics 

• The metrics are not on the same absolute scale and therefore are difficult to compare or 
combine 

To overcome these limitations, normalization factors were developed. The design evaluation cri- 
teria established earlier were used to formulate a normalization equation for each of the measur- 
able design characteristics. (Some of the normalization equations and their rationales are pre- 
sented in Table 2.) An absolute measure ranging from 0 to I, with 1 meaning most desirable, 
was developed for each factor measured by the Design Metrics Evaluator. This allowed normalized 
metrics to be combined to provide absolute measures for design characteristics at various levels 
of the hierarchy. Tite calcuiaiion of the simplicity metric by combining its components is also 
illustrated in Table 2 . 
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The importance of different design characteristics can vary for different applications. The overall 
quality of a program’s design can be obtained by combining characteristics of interest to that pro- 
gram, weighted by their importance. The assignment of weights to the various design characteris- 
tics can be subjective and can vary according to application. 


THE DESIGN METRICS EVALUATOR 

The Design Metrics Evaluator is an automated software tool consisting of three modules that in- 
terface through a global data base. The Metrics Module collects the design metrics during parsing 
and stores the results in the global data base. The Normet Module normalizes the metrics. The 
Display Module displays the resultant metrics, in normalized or unnormalized form. 

The Design Metrics Evaluator has the capability of collecting design metrics for a procedure, a 
module, or a complete program. It is a user-oriented tool with six user-selectable options: 

• Collection of design metrics 

• Listing of the prenormalized design metrics 

• Normalization of the design metrics 

• Output of normalized design metrics to a file 

• Output of normalized design metrics to the printer 

• Collection of program statistics 

Design metrics for a module or a program can be calculated with user-specified weights assigned 
to the design characterisiics. This capability allows weights to be tailored or refined for a specific 
software application. 

Figure 2 presents the normalized design metrics for one module of a software tool. Shown at the 
top are the metrics for three groups of attributes (simplicity, structuredness, and readability) for 
the procedures identified in the left-most column. The parenthetical numbers are the standard 
deviations for ihc module. These results indicate that, in general, the procedures of this module 
are of reasonable length (program length metric = 0.766). The low statement size metric of 
0.289 indicates that the statements are too long. This is in general true for all the procedures, as 
indicated by the standard deviation of 0.186 and the statement size metric for each procedure. 
The flow-interruption metric of 1.0 results because there are no unconditional branches in any of 
the procedures. 

Absolute measures on selected design characteristics for the module are shown at the bottom of 
Figure 2. These metrics were obtained by combining the values of their supporting factors, 
weighted by importance. For example, the simplicity metrics were calculated with equal weight- 
ings and the sufficiency metrics were calculated with simplicity weighted twice as much as 
structuredness: 

Simplicity = 1/4 (program length + statement size + decision count + statement nesting 
level) 

= 1/4 (0.766 + 0.289 + 0.614 + 0.896) = 0.641 
Sufficiencv = 1/3 (2 simplicity + structuredness) 

= (0(0.641) + 0.814) = 0.699 
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The summary metrics suggest that the module is fairly well designed. The lowest value, 0.641, 
for the simplicity metrics reflects the large statement size and points out that this area needs to 
be improved. Furthermore, the maintainability metrics are higher than the usability metrics, 
pointing out that the design is weak in the usability area. 


FUTURE WORK 

The current Design Metrics Evaluator supplies a program designer with objective metrics and sta- 
tistics based on design language constructs. Although many design characteristics are not quanti- 
tatively measurable, the DME output metrics provide a guideline for determining design quality. 

Further work is being considered in the following areas; 

• Development of systematic approach to subjective software design evaluation 

• Incorporation of design metrics for data structures 

• Incorporation of selected complexity metrics found in the literature 

Further validation of the Design Metrics Evaluator is another area for future work. The tool has 
been tested on a number of different software application programs but should be tested more 
extensively before release for general use. 
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Table 1 

Candidate Software Characteristics 

Acceptability 

Generality 

Accessibility 

Hierarchicality 

Accountability 

Human Factors 

Accuracy 

Improvability 

Adaptability 

Integrity 

Augmentability 

Intelligibility 

Brevity 

Interoperability 

Changeability 

Legibility 

Clarity 

Linearity 

Communicativeness 

Machine-Independence 

Compatibility 

Maintainability 

Completeness 

Meaningfulness 

Comprehensibility 

Modifiability 

Conciseness 

Modularity 

Confirrnability 

Operability 

Consistency 

Ordered ness 

Convenience 

Portability 

Correctness 

Precision 

Expandability 

Readability 

Expressiveness 

Reliability 

Extensibility 

Repairability 

Flexibility 

Reusability 


Robustness 

Ruggedness 

Security 

Self-Containedness 

Self-Descriptiveness 

Serviceability 

Simplicity 

Stability 

Structuredness 

Succinctness 

Sufficiency 

Terseness 

Testability 

Tolerance 

Transferability 

Understandability 

Uniformity 

Usability 

User-Centeredness 

Validity 

Verifiability 
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Figure 1 Hierarchical structure of good software design characteristics. 
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Table 2. NormaUzation Equations for the Components of Simplicity 
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Figure 2. Listing of Normalized Metrics for a Sample Module and Its Procedures 
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SIMPLICITY; 

0.641 

STRUCTUREDNESS; 

0.814 

READRBIL'TY; 

0.909 

SUFFICIENCY; 

0.699 

RELIABILITY; 

0.699 

USABILITY; 

0.699 

SUCCINCTNESS; 

0.992 

UNDERSTANDABILITY; 

0.912 

VERIFIABILITY; 

0.892 

MODIFIABILITY; 

0.814 

MAINTAINABILITY; 

0.873 
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